Oct 31 00:02:05.493181 ip-10-0-137-148 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:02:05.911087 ip-10-0-137-148 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:02:05.911087 ip-10-0-137-148 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:02:05.911087 ip-10-0-137-148 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:02:05.911087 ip-10-0-137-148 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921252    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921329    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921335    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921339    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921342    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921346    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.921340 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921352    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921361    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921365    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921369    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921372    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921376    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921380    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921387    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921391    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921395    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921399    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921403    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921407    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921410    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921414    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921418    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921422    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921426    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921430    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921434    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.921716 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921439    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921442    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921447    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921451    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921456    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921460    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921464    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921468    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921475    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921482    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921486    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921492    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921497    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921502    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921507    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921511    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921516    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921520    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921525    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.922569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921530    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921534    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921538    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921542    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921547    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921551    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921555    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921560    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921564    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921568    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921573    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921577    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921581    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921586    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921590    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921594    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921599    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921603    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921607    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921611    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.923374 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921615    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921620    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921624    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921628    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921632    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921637    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921641    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921646    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921650    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921655    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921660    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921665    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921669    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921674    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921678    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921682    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921687    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921691    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921695    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921699    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.924252 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921703    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921708    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921711    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921715    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921719    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921723    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921728    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921732    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921736    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921740    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921745    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921749    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921753    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921762    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921767    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921771    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921775    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921783    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921789    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.925096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921795    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.921803    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923136    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923147    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923156    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923163    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923168    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923172    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923177    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923182    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923186    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923191    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923196    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923200    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923204    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923208    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923212    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923216    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923221    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.925613 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923226    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923230    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923234    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923238    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923242    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923246    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923250    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923254    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923259    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923264    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923268    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923273    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923277    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923282    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923286    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923290    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923295    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923299    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923303    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.926472 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923308    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923312    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923316    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923320    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923324    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923328    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923333    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923337    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923341    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923345    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923349    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923354    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923358    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923362    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923366    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923370    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923375    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923378    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923382    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.927298 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923387    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923391    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923395    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923401    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923405    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923409    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923414    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923418    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923422    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923427    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923431    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923435    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923439    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923443    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923447    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923451    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923466    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923470    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923474    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923478    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.927978 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923482    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923487    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923491    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923498    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923506    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923511    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923515    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923519    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923523    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923528    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923532    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923537    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923542    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923546    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923550    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923555    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923560    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923564    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923568    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923573    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.928657 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923577    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923581    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923585    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923589    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923594    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923597    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923601    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923605    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923611    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923616    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.923620    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924446    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924455    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924460    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924464    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924469    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924474    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924478    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924483    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924487    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.929194 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924492    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924496    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924500    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924504    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924509    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924513    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924518    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924522    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924528    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924532    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924537    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924541    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924545    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924549    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924553    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924557    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924562    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924566    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.929727 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924571    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924577    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924583    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924589    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924594    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924598    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924602    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924607    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924611    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924615    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924620    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924624    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924628    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924632    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924636    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924640    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924644    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924648    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924653    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.930210 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924657    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924662    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924666    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924670    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924677    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924682    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924686    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924690    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924694    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924698    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924702    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924707    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924711    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924715    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924719    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924723    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924727    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924731    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924736    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924740    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.930695 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924744    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924748    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924752    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924757    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924761    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924765    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924769    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924774    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924778    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924782    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924787    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924791    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924795    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924799    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924803    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924807    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924811    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924816    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924821    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924825    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.931219 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924829    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924833    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924837    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924841    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924845    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924850    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924854    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924858    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924863    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924867    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924871    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924875    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924879    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924883    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924887    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924891    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924896    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924920    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924926    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.924930    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.931674 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925481    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925494    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925506    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925513    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925520    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925525    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925532    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925539    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925544    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925549    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925556    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925561    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925566    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925571    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925575    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925580    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925585    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925589    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925595    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925600    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925605    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925610    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925615    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925621    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:02:05.932176 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925625    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925630    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925638    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925643    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925648    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925653    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925658    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925663    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925670    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925674    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925679    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925684    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925689    2574 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925694    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925701    2574 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925705    2574 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925710    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925715    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925720    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925730    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925735    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925740    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925745    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925750    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925755    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:02:05.932820 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925759    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925765    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925770    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925775    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925780    2574 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925786    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925791    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925796    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925801    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925808    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925813    2574 flags.go:64] FLAG: --help="false"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925818    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925824    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925829    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925834    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925839    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925844    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925849    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925854    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925858    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925863    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925868    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925873    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925877    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:02:05.933453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925882    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925886    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925893    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925918    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925923    2574 flags.go:64] FLAG: --lock-file=""
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925928    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925933    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925938    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925948    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925952    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925957    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925962    2574 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925967    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925972    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925977    2574 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925981    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925988    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.925995    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926001    2574 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926006    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926011    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926016    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926020    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926025    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926030    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:02:05.934394 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926035    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926046    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926051    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926056    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926061    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926065    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926070    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926075    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926079    2574 flags.go:64] FLAG: --port="10250"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926085    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926090    2574 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-094cae1e0a98e38aa"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926097    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926101    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926106    2574 flags.go:64] FLAG: --register-node="true"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926111    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926118    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926122    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926126    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926131    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926137    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926142    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926147    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926152    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926157    2574 flags.go:64] FLAG: --runonce="false"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926163    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926168    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926173    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:02:05.935029 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926177    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926182    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926187    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926192    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926197    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926202    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926207    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926212    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926216    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926221    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926226    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926230    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926239    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926244    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926250    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926255    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926260    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926264    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926269    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926274    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926286    2574 flags.go:64] FLAG: --v="2"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926293    2574 flags.go:64] FLAG: --version="false"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926299    2574 flags.go:64] FLAG: --vmodule=""
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926305    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.926310    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:02:05.935660 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926526    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926533    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926539    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926544    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926550    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926554    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926559    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926563    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926568    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926574    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926580    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926586    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926591    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926596    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926600    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926605    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926609    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926614    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926618    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.936295 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926622    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926626    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926633    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926637    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926641    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926645    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926650    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926654    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926659    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926663    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926670    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926675    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926680    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926684    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926689    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926694    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926698    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926705    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926709    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.936777 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926713    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926718    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926722    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926726    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926730    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926734    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926739    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926743    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926747    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926751    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926755    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926759    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926764    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926768    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926772    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926777    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926781    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926785    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926790    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926794    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.937341 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926799    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926804    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926808    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926812    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926816    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926820    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926824    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926829    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926833    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926837    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926843    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926847    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926851    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926855    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926859    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926863    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926868    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926872    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926877    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.937821 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926881    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926885    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926889    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926893    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926915    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926919    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926924    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926928    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926934    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926939    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926942    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926946    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926951    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926955    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926961    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926965    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926969    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926973    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926977    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926981    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.938303 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926986    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926990    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.926994    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927001    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927005    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927009    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927013    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927018    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927022    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.927065    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927123    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927129    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927134    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927139    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927143    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927147    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.938780 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927152    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927156    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927161    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927165    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927169    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927174    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927179    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927183    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927188    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927192    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927196    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927201    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927205    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927210    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927214    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927219    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927223    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927227    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927231    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927238    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.939223 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927244    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927249    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927254    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927259    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927264    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927268    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927273    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927277    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927281    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927285    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927290    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927294    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927298    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927302    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927306    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927310    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927314    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927318    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927323    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.939706 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927327    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927334    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927341    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927346    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927350    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927355    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927360    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927366    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927370    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927375    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927380    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927384    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927390    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927395    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927398    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927402    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927407    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927411    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927415    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927420    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.940168 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927424    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927428    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927432    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927437    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927440    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927445    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927449    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927453    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927457    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927461    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927466    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927470    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927474    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927478    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927482    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927486    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927491    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927495    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927500    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.940642 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927504    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927509    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927513    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927517    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927521    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927527    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927531    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927535    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927539    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927544    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927548    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927552    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927556    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927562    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927566    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927570    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927574    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927578    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927583    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.941103 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927587    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927591    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.927595    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.935503    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.935518    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.936728    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.936735    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936795    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936801    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936804    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936808    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936811    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936814    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936817    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936819    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936822    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936826    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936830    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936834    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936837    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.941560 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936840    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936843    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936846    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936849    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936851    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936854    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936857    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936859    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936862    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936864    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936866    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936869    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936872    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936874    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936878    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936880    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936883    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936886    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936889    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936891    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.942090 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936894    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936896    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936918    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936921    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936924    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936926    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936929    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936932    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936935    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936937    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936940    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936942    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936945    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936948    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936950    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936953    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936955    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936958    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936961    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936963    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.942569 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936966    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936968    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936970    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936973    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936975    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936978    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936981    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936983    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936987    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936989    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936992    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936994    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936996    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.936999    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937002    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937004    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937006    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937009    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937011    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.943157 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937014    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937017    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937019    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937022    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937024    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937027    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937030    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937032    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937034    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937037    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937039    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937042    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937044    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937047    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937049    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937052    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937054    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937057    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937060    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.943597 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937062    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937065    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937067    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937070    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937072    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937075    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937077    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937080    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937084    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937088    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937091    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937093    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937096    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937099    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937102    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.944068 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.937131    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937266    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937271    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937274    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937277    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937280    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937283    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937286    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937288    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937291    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937293    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937296    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937298    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937301    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937303    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937306    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937308    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937311    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937314    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937316    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.944423 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937318    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937321    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937323    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937326    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937328    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937331    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937333    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937336    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937338    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937342    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937346    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937350    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937352    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937355    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937358    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937360    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937363    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937365    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937368    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937371    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.944888 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937373    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937376    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937378    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937381    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937384    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937386    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937389    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937391    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937394    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937396    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937398    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937401    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937403    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937406    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937408    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937411    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937413    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937415    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937418    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937420    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.945413 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937423    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937425    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937427    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937430    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937432    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937435    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937437    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937440    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937442    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937445    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937447    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937450    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937453    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937455    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937458    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937461    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937463    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937466    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937468    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.945885 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937470    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937473    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937476    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937478    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937481    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937483    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937486    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937489    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937491    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937494    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937496    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937499    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937501    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937504    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937506    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937508    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937511    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937514    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937516    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.946359 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937519    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937521    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937525    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937529    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937532    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937535    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937538    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937541    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937544    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.937567    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937600    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937604    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937607    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937610    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937613    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:05.946794 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937615    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937618    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937620    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937623    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937625    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937628    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937630    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937633    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937635    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937638    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937640    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937642    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937645    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937647    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937649    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937652    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937654    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937657    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937660    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937662    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:05.947176 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937664    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937667    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937669    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937671    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937674    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937677    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937679    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937681    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937684    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937687    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937689    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937692    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937694    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937697    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937699    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937708    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937711    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937713    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937716    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:05.947635 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937718    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937720    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937723    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937725    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937728    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937730    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937733    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937736    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937738    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937741    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937743    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937746    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937748    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937750    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937753    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937755    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937757    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937760    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937762    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:05.948096 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937766    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937768    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937771    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937775    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937778    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937781    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937784    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937787    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937789    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937793    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937796    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937798    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937800    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937803    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937805    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937808    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937810    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937813    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937816    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:05.948532 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937818    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937821    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937823    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937826    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937829    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937832    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937834    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937837    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937839    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937842    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937844    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937847    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937849    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937852    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937855    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937857    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937860    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937862    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937864    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937869    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:05.949022 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937873    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937876    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937879    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:05.937882    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.937951    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.941351    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.942266    2574 server.go:1014] "Starting client certificate rotation"
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.942361    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.942401    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.944131    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:02:05.949493 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.945254    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:02:05.958093 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.958080    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:02:05.962334 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.962318    2574 log.go:25] "Validated CRI v1 image API"
Oct 31 00:02:05.963092 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.963059    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:02:05.965485 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.965460    2574 fs.go:135] Filesystem UUIDs: map[34186ae9-c764-48a7-8bf7-0b175261029f:/dev/nvme0n1p3 48116720-39b3-4a68-8baa-c9ff8aab0ded:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 00:02:05.965544 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.965486    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 31 00:02:05.972912 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.972785    2574 manager.go:217] Machine: {Timestamp:2026-10-31 00:02:05.971529365 +0000 UTC m=+0.356175173 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099557 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a751e8bdb771810dd8dfca4615406 SystemUUID:ec2a751e-8bdb-7718-10dd-8dfca4615406 BootID:e646b4b4-9bf8-448a-bc93-caff5599036c Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:48:ac:79:97:b7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:48:ac:79:97:b7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:62:28:89:bf:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:02:05.976688 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.976676    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:02:05.976778 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.976762    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:02:05.977784 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.977758    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:02:05.977988 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.977786    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-148.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:02:05.978933 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.978921    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:02:05.978971 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.978935    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:02:05.978971 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.978959    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:02:05.979022 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.978980    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:02:05.979068 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.979059    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:02:05.981716 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.981704    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:02:05.982714 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.982678    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:02:05.983588 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.983566    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:02:05.983666 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.983613    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:02:05.983666 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.983631    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:02:05.983666 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.983642    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:02:05.983666 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.983656    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:02:05.984187 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.984170    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:02:05.987336 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.987319    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:02:05.991599 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.991586    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:02:05.992320 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992308    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:02:05.992355 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992339    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:02:05.992497 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992487    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:02:05.992532 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992502    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:02:05.992532 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992509    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:02:05.992532 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992514    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:02:05.992532 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992521    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:02:05.992532 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.992526    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:02:05.993191 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993182    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:02:05.993220 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993192    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:02:05.993220 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993200    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:02:05.993220 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993205    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:02:05.993220 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993214    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:02:05.993341 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993222    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:02:05.993787 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993779    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:02:05.993819 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.993788    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:02:05.997575 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.997563    2574 server.go:1263] "Started kubelet"
Oct 31 00:02:05.997687 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.997642    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:02:05.997734 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.997656    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:02:05.997734 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.997708    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:02:05.998300 ip-10-0-137-148 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:02:05.999200 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:05.999184    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:02:06.001560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.001546    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:02:06.003062 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003042    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:02:06.003156 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003067    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:02:06.004814 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003583    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:02:06.004814 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003586    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:02:06.004814 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003644    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:02:06.004814 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003745    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:02:06.004814 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.003751    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:02:06.005199 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.005176    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:06.005292 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005261    2574 factory.go:55] Registering systemd factory
Oct 31 00:02:06.005292 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005278    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:02:06.005560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005543    2574 factory.go:153] Registering CRI-O factory
Oct 31 00:02:06.005659 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005644    2574 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:02:06.005725 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.005692    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 31 00:02:06.005725 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005707    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:02:06.005799 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005731    2574 factory.go:103] Registering Raw factory
Oct 31 00:02:06.005799 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.005746    2574 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:02:06.006546 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.006530    2574 manager.go:319] Starting recovery of all containers
Oct 31 00:02:06.017150 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.017116    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:02:06.018624 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.018602    2574 manager.go:324] Recovery completed
Oct 31 00:02:06.019891 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.019865    2574 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Oct 31 00:02:06.022797 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.022777    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.024677 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.024662    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.024749 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.024694    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.024749 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.024707    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.025215 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.025203    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:02:06.025215 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.025209    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.025215 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.025213    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:02:06.025332 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.025231    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:02:06.026167 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.026153    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.026208 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.026179    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.026208 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.026188    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.026259 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.026236    2574 policy_none.go:50] "Start"
Oct 31 00:02:06.026259 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.026253    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:02:06.026333 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.026264    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:02:06.027315 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.027304    2574 policy_none.go:44] "Start"
Oct 31 00:02:06.059615 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.059588    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.068970    2574 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.069003    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.069013    2574 server.go:87] "Starting device plugin registration server"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.069238    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.069248    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.069363    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.069446    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.069456    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.070005    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.070079    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.079926    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.080042    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.080058    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:02:06.081267 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.080087    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 31 00:02:06.146610 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.146568    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-148.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 31 00:02:06.147543 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.146596    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.us-west-1.compute.internal.18e374474084cf41  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:05.997535041 +0000 UTC m=+0.382180849,LastTimestamp:2026-10-31 00:02:05.997535041 +0000 UTC m=+0.382180849,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.us-west-1.compute.internal,}"
Oct 31 00:02:06.170367 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.170310    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.171294 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.171281    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.171380 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.171307    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.171380 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.171318    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.171380 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.171344    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.180845 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.180824    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"]
Oct 31 00:02:06.180951 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.180886    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.181659 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.181645    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.181707 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.181675    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.181707 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.181684    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.182847 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.182835    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.183020 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183007    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.183067 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183035    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.183581 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183562    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.183581 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183572    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.183727 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183592    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.183727 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183597    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.183727 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183606    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.183727 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.183607    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.184924 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.184888    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.185009 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.184931    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.185587 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.185572    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.185674 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.185593    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.185674 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.185603    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.194673 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.194654    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-148.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 31 00:02:06.205693 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.205677    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.205765 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.205703    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.205765 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.205720    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3031cca7be5520d5a22eb64a065c3c72-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"3031cca7be5520d5a22eb64a065c3c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.220016 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.219994    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:02:06.221702 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.221683    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.225205 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.225188    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.244422 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.244400    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 31 00:02:06.244422 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.244400    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-148.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 31 00:02:06.244561 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.244382    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.us-west-1.compute.internal.18e374474222f9e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-148.us-west-1.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:06.02467786 +0000 UTC m=+0.409323668,LastTimestamp:2026-10-31 00:02:06.02467786 +0000 UTC m=+0.409323668,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.us-west-1.compute.internal,}"
Oct 31 00:02:06.244561 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.244501    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 31 00:02:06.274776 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.274754    2574 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hf4p"
Oct 31 00:02:06.287516 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.287498    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5hf4p"
Oct 31 00:02:06.305648 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.305620    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 31 00:02:06.305740 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.305651    2574 kubelet_node_status.go:115] "Unable to register node with API server, error getting existing node" err="nodes \"ip-10-0-137-148.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"nodes\" in API group \"\" at the cluster scope" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.305801 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.305709    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-148.us-west-1.compute.internal.18e3744742235412  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-137-148.us-west-1.compute.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:06.024700946 +0000 UTC m=+0.409346753,LastTimestamp:2026-10-31 00:02:06.024700946 +0000 UTC m=+0.409346753,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.us-west-1.compute.internal,}"
Oct 31 00:02:06.305985 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.305966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3031cca7be5520d5a22eb64a065c3c72-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"3031cca7be5520d5a22eb64a065c3c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.306059 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.306000    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.306059 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.306030    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3031cca7be5520d5a22eb64a065c3c72-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"3031cca7be5520d5a22eb64a065c3c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.306059 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.306049    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.306161 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.306025    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.306161 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.306057    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.498745 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.498662    2574 nodelease.go:50] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-148.us-west-1.compute.internal\" not found" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.505925 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.505895    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:06.506858 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.506840    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:06.506971 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.506868    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:06.506971 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.506878    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:06.506971 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.506924    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.524959 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.524938    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.527689 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.527665    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.564560 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.564529    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:06.564560 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.564564    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-148.us-west-1.compute.internal\": node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:06.685110 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.685050    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:06.785507 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.785418    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:06.886336 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.886299    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:06.942731 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:06.942686    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 31 00:02:06.943193 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.942863    2574 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aec41e53b5c24408ea18b6975af3dda9-0bfa0ebd3c963d0d.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-137-148.us-west-1.compute.internal.18e3744742237c4b\": read tcp 10.0.137.148:44850->3.133.20.147:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-137-148.us-west-1.compute.internal.18e3744742237c4b  default   4693 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-148.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-137-148.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-137-148.us-west-1.compute.internal,},FirstTimestamp:2026-10-31 00:02:06 +0000 UTC,LastTimestamp:2026-10-31 00:02:06.171322472 +0000 UTC m=+0.555968279,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-148.us-west-1.compute.internal,}"
Oct 31 00:02:06.987230 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:06.987208    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.087805 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:07.087738    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.188811 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:07.188784    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.289842 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:07.289815    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.289842 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.289830    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:57:07 +0000 UTC" deadline="2028-05-27 20:12:24.911897034 +0000 UTC"
Oct 31 00:02:07.290016 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.289852    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13796h10m17.622047919s"
Oct 31 00:02:07.306234 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.306216    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:07.307264 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.307246    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:07.307405 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.307282    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:07.307405 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.307299    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:07.343308 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.343244    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:07.390362 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:07.390338    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.415063 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.415030    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:07.491497 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:07.491469    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.584446 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:07.584411    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aae722a65c2075f78dd611b7e7f739.slice/crio-bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8 WatchSource:0}: Error finding container bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8: Status 404 returned error can't find the container with id bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8
Oct 31 00:02:07.587390 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:07.587356    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3031cca7be5520d5a22eb64a065c3c72.slice/crio-e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05 WatchSource:0}: Error finding container e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05: Status 404 returned error can't find the container with id e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05
Oct 31 00:02:07.591645 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:07.591622    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:07.592349 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.592335    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:02:07.628058 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.628036    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:07.703236 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.703209    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:07.765555 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.765533    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:02:07.768039 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.768026    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:07.785957 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:07.785937    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:08.877855 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:08.877836    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:02:09.037487 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.037414    2574 apiserver.go:52] "Watching apiserver"
Oct 31 00:02:09.056463 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.056441    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 31 00:02:09.136544 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.136493    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"3031cca7be5520d5a22eb64a065c3c72","Type":"ContainerStarted","Data":"e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05"}
Oct 31 00:02:09.137499 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.137478    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"f7aae722a65c2075f78dd611b7e7f739","Type":"ContainerStarted","Data":"bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8"}
Oct 31 00:02:09.142374 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.142357    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:02:09.142669 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.142649    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-chjbs","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal","openshift-multus/multus-additional-cni-plugins-9b5br","openshift-multus/network-metrics-daemon-9cbv5","openshift-network-diagnostics/network-check-target-s64vg","openshift-ovn-kubernetes/ovnkube-node-s95vh","kube-system/konnectivity-agent-cprhd","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4","openshift-image-registry/node-ca-f9k85","openshift-multus/multus-pqz2l","kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal","openshift-cluster-node-tuning-operator/tuned-mnzps","kube-system/global-pull-secret-syncer-gfzr6","openshift-network-operator/iptables-alerter-9bwpb"]
Oct 31 00:02:09.144487 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.144472    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.145463 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.145444    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.146555 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.146542    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.147452 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.147440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.147522 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.147489    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:09.148456 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.148440    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:09.148530 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.148508    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:09.149559 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.149544    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.150624 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.150608    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.152047 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.152032    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.153149 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.153133    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.154468 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.154450    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.155728 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.155707    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.155823 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.155775    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:09.157058 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.157041    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.168837 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168820    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-slash\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.168910 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-node-log\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.168910 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168885    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-bin\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.168910 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168906    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-netd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168921    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-device-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168941    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d05191cb-2a10-4929-8000-b87e80b7070f-serviceca\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168964    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19be463a-c645-48ab-a633-42fb816ad71c-iptables-alerter-script\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168982    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gp4rs\" (UniqueName: \"kubernetes.io/projected/19be463a-c645-48ab-a633-42fb816ad71c-kube-api-access-gp4rs\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.168995    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-kubelet-config\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169009    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-systemd\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.169025 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169037    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169061    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169086    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-modprobe-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169114    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-sys\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169136    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169158    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169181    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-bin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169221    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-hostroot\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169255    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-conf-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.169298 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvwng\" (UniqueName: \"kubernetes.io/projected/d05191cb-2a10-4929-8000-b87e80b7070f-kube-api-access-gvwng\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-kubelet\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169326    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-netns\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169350    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5z6b\" (UniqueName: \"kubernetes.io/projected/433c6858-00d2-47fc-8e98-9df265e3ce28-kube-api-access-s5z6b\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169374    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-socket-dir-parent\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169416    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cnibin\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169439    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169465    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nw9mm\" (UniqueName: \"kubernetes.io/projected/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-kube-api-access-nw9mm\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a873a458-b1cc-4b84-9a61-a35b5c783868-konnectivity-ca\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169549    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169563    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-k8s-cni-cncf-io\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169578    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8244x\" (UniqueName: \"kubernetes.io/projected/f14b5c6b-3af4-49b0-a5bb-930d814fa069-kube-api-access-8244x\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169601    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a873a458-b1cc-4b84-9a61-a35b5c783868-agent-certs\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169622    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8tq\" (UniqueName: \"kubernetes.io/projected/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kube-api-access-fd8tq\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169640    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-lib-modules\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.169698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169657    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-host\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169675    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169718    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-cnibin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169754    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-script-lib\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169783    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169809    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-config\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169845    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-system-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-netns\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169899    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-daemon-config\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169914    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05191cb-2a10-4929-8000-b87e80b7070f-host\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169947    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169973    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-etc-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.169998    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f14b5c6b-3af4-49b0-a5bb-930d814fa069-hosts-file\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170020    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-tmp\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170044    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170069    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-kubernetes\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-cni-binary-copy\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170115    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-multus-certs\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170138    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-ovn\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-run\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170168    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-var-lib-kubelet\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170182    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170201    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19be463a-c645-48ab-a633-42fb816ad71c-host-slash\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170217    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-systemd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170232    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-env-overrides\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170247    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysconfig\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170282    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-conf\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170305    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-kubelet\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170321    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-systemd-units\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170342    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovn-node-metrics-cert\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170357    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf5sc\" (UniqueName: \"kubernetes.io/projected/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-kube-api-access-rf5sc\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170373    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.170803 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170414    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-system-cni-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170438    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-log-socket\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170454    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvb88\" (UniqueName: \"kubernetes.io/projected/1a6de299-a94e-4591-acac-b5018a133c65-kube-api-access-lvb88\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170469    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-multus\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170531    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-var-lib-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170565    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-dbus\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170637    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170661    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14b5c6b-3af4-49b0-a5bb-930d814fa069-tmp-dir\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170683    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170706    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-tuned\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170731    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-etc-kubernetes\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.171231 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.170755    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttttp\" (UniqueName: \"kubernetes.io/projected/bc469038-9017-4941-972e-0cb0bc3adf6c-kube-api-access-ttttp\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.199040 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199023    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.199040 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199032    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.199196 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199175    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:02:09.199335 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199322    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:02:09.199401 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199320    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gvgd9\""
Oct 31 00:02:09.199401 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199320    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgjmv\""
Oct 31 00:02:09.199505 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.199492    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt67z\""
Oct 31 00:02:09.200596 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.200583    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:02:09.201539 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.201526    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.201590 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.201563    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.201590 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.201579    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:02:09.201715 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.201704    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:02:09.211540 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211519    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ddpr\""
Oct 31 00:02:09.211608 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211548    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.211719 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211706    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.211791 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211780    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsk29\""
Oct 31 00:02:09.211847 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211829    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.211902 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211887    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.211952 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.211931    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.212019 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.212007    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:02:09.212055 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.212036    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:02:09.246563 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.246541    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:02:09.246642 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.246576    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.246642 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.246586    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:02:09.246708 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.246686    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:02:09.248829 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.248817    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27h5h\""
Oct 31 00:02:09.257299 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.257285    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:02:09.271502 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271481    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-systemd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271621 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-env-overrides\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271621 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271524    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysconfig\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.271621 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271544    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-conf\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.271621 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271568    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-kubelet\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271633    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-kubelet\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271643    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-systemd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271675    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-conf\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271684    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-systemd-units\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271705    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovn-node-metrics-cert\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271706    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-systemd-units\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5sc\" (UniqueName: \"kubernetes.io/projected/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-kube-api-access-rf5sc\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.271750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271743    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271686    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysconfig\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271769    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-system-cni-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-log-socket\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271815    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-system-cni-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271838    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-log-socket\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.271852    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvb88\" (UniqueName: \"kubernetes.io/projected/1a6de299-a94e-4591-acac-b5018a133c65-kube-api-access-lvb88\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.271953    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret podName:660c1701-2942-48bd-aee9-ff5341e6ffce nodeName:}" failed. No retries permitted until 2026-10-31 00:02:09.771910405 +0000 UTC m=+3.104076930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret") pod "global-pull-secret-syncer-gfzr6" (UID: "660c1701-2942-48bd-aee9-ff5341e6ffce") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-multus\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271989    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-env-overrides\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272031    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-var-lib-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.271992    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272054    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-multus\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272058    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-dbus\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272099    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272103    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-var-lib-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14b5c6b-3af4-49b0-a5bb-930d814fa069-tmp-dir\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272156    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272185    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-tuned\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272208    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-etc-kubernetes\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272228    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-dbus\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272232    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttttp\" (UniqueName: \"kubernetes.io/projected/bc469038-9017-4941-972e-0cb0bc3adf6c-kube-api-access-ttttp\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272245    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-slash\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272277    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272288    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-etc-kubernetes\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-node-log\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272320    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-slash\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.272566 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272336    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-node-log\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272322    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-bin\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272359    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-bin\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272375    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-netd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272412    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-netd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272418    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-device-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272440    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d05191cb-2a10-4929-8000-b87e80b7070f-serviceca\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272448    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14b5c6b-3af4-49b0-a5bb-930d814fa069-tmp-dir\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19be463a-c645-48ab-a633-42fb816ad71c-iptables-alerter-script\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272491    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4rs\" (UniqueName: \"kubernetes.io/projected/19be463a-c645-48ab-a633-42fb816ad71c-kube-api-access-gp4rs\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272499    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-device-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272517    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-kubelet-config\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272542    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-systemd\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272564    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-kubelet-config\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272581    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272606    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-systemd\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.273393 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272638    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272659    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-modprobe-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272678    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-sys\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272696    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272716    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-bin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272745    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-hostroot\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-conf-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272810    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwng\" (UniqueName: \"kubernetes.io/projected/d05191cb-2a10-4929-8000-b87e80b7070f-kube-api-access-gvwng\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272829    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-kubelet\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272849    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-netns\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272854    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-sys\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272862    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-modprobe-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272871    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z6b\" (UniqueName: \"kubernetes.io/projected/433c6858-00d2-47fc-8e98-9df265e3ce28-kube-api-access-s5z6b\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272896    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-conf-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272905    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-socket-dir-parent\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.274144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272928    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272934    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cnibin\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272959    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cnibin\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272966    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.272830    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273065    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19be463a-c645-48ab-a633-42fb816ad71c-iptables-alerter-script\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273078    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-hostroot\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273083    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-kubelet\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273121    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-bin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273129    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-socket-dir-parent\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273176    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273173    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-netns\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273182    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9mm\" (UniqueName: \"kubernetes.io/projected/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-kube-api-access-nw9mm\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273327    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273369    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a873a458-b1cc-4b84-9a61-a35b5c783868-konnectivity-ca\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273527    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-k8s-cni-cncf-io\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273534    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273557    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8244x\" (UniqueName: \"kubernetes.io/projected/f14b5c6b-3af4-49b0-a5bb-930d814fa069-kube-api-access-8244x\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273585    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a873a458-b1cc-4b84-9a61-a35b5c783868-agent-certs\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273587    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-k8s-cni-cncf-io\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273612    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8tq\" (UniqueName: \"kubernetes.io/projected/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kube-api-access-fd8tq\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273636    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-lib-modules\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273661    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-host\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273683    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273697    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273712    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-cnibin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273740    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-script-lib\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273765    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273819    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-lib-modules\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273827    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-config\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273877    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-system-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-netns\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.275655 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273926    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-daemon-config\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05191cb-2a10-4929-8000-b87e80b7070f-host\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273977    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.273996    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a873a458-b1cc-4b84-9a61-a35b5c783868-konnectivity-ca\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274002    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-etc-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274044    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-etc-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274052    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f14b5c6b-3af4-49b0-a5bb-930d814fa069-hosts-file\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274079    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-tmp\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274152    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f14b5c6b-3af4-49b0-a5bb-930d814fa069-hosts-file\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274160    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274196    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-kubernetes\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274214    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274246    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-cni-binary-copy\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274269    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-system-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274272    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-multus-certs\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-config\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274299    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-ovn\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274306    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-netns\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274321    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-run\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274345    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-var-lib-kubelet\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274362    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05191cb-2a10-4929-8000-b87e80b7070f-host\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.274441    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274445    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274483    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-kubernetes\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.274538    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:09.774523077 +0000 UTC m=+3.106689592 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.274953    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-script-lib\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275026    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275041    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-cni-binary-copy\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275097    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-host\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275112    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-multus-certs\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275133    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19be463a-c645-48ab-a633-42fb816ad71c-host-slash\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275220    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-ovn\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275235    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19be463a-c645-48ab-a633-42fb816ad71c-host-slash\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.276703 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275234    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-var-lib-kubelet\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275036    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275291    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275311    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-run\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.275353    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-cnibin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.276321    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-tuned\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.276463    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovn-node-metrics-cert\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.276613    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-tmp\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.277144 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.276649    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a873a458-b1cc-4b84-9a61-a35b5c783868-agent-certs\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.297557 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.297516    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 31 00:02:09.297557 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.297549    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-6hwz7"
Oct 31 00:02:09.299913 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.299895    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.299913 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.299906    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:02:09.300016 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.299922    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:02:09.300078 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300060    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:02:09.300177 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300064    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqqqs\""
Oct 31 00:02:09.300177 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300069    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.300177 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300100    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:02:09.300177 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300144    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wr2gb\""
Oct 31 00:02:09.300307 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300176    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:02:09.300307 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.300192    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7xfz6\""
Oct 31 00:02:09.303329 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.303310    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d05191cb-2a10-4929-8000-b87e80b7070f-serviceca\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.305553 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.305539    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-daemon-config\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.333914 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.333892    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:02:09.333914 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.333908    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:02:09.333914 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.333917    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9zp2t for pod openshift-network-diagnostics/network-check-target-s64vg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:09.334088 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.334021    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t podName:49220507-cecd-48d6-b19a-3f8479146036 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:09.834007192 +0000 UTC m=+3.166173723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9zp2t" (UniqueName: "kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t") pod "network-check-target-s64vg" (UID: "49220507-cecd-48d6-b19a-3f8479146036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:09.335869 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.335847    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5sc\" (UniqueName: \"kubernetes.io/projected/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-kube-api-access-rf5sc\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.335933 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.335887    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwng\" (UniqueName: \"kubernetes.io/projected/d05191cb-2a10-4929-8000-b87e80b7070f-kube-api-access-gvwng\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.343951 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.342446    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvb88\" (UniqueName: \"kubernetes.io/projected/1a6de299-a94e-4591-acac-b5018a133c65-kube-api-access-lvb88\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.343951 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.343145    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8tq\" (UniqueName: \"kubernetes.io/projected/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kube-api-access-fd8tq\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.343951 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.343229    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttttp\" (UniqueName: \"kubernetes.io/projected/bc469038-9017-4941-972e-0cb0bc3adf6c-kube-api-access-ttttp\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.346626 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.346607    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4rs\" (UniqueName: \"kubernetes.io/projected/19be463a-c645-48ab-a633-42fb816ad71c-kube-api-access-gp4rs\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.346698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.346647    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z6b\" (UniqueName: \"kubernetes.io/projected/433c6858-00d2-47fc-8e98-9df265e3ce28-kube-api-access-s5z6b\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.346698 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.346663    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9mm\" (UniqueName: \"kubernetes.io/projected/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-kube-api-access-nw9mm\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.347356 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.347338    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8244x\" (UniqueName: \"kubernetes.io/projected/f14b5c6b-3af4-49b0-a5bb-930d814fa069-kube-api-access-8244x\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.455145 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.455118    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:09.461598 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.461578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:09.463489 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.463466    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda873a458_b1cc_4b84_9a61_a35b5c783868.slice/crio-562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef WatchSource:0}: Error finding container 562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef: Status 404 returned error can't find the container with id 562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef
Oct 31 00:02:09.469346 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.469322    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod433c6858_00d2_47fc_8e98_9df265e3ce28.slice/crio-bb97eb920df58435d66cf84560fa183015f990a9e7f16f44132ff310ef754fe7 WatchSource:0}: Error finding container bb97eb920df58435d66cf84560fa183015f990a9e7f16f44132ff310ef754fe7: Status 404 returned error can't find the container with id bb97eb920df58435d66cf84560fa183015f990a9e7f16f44132ff310ef754fe7
Oct 31 00:02:09.473154 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.473135    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:09.480106 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.479485    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:09.480106 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.479539    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:09.481493 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.481451    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c6eb6e_5d1c_4f9f_9e44_1dc0645f6fa2.slice/crio-aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62 WatchSource:0}: Error finding container aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62: Status 404 returned error can't find the container with id aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62
Oct 31 00:02:09.484729 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.484699    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:09.490595 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.490575    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:09.490873 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.490853    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9cde72b_3aa1_4c45_95e1_72a55236dc1e.slice/crio-371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1 WatchSource:0}: Error finding container 371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1: Status 404 returned error can't find the container with id 371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1
Oct 31 00:02:09.492711 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.492655    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19be463a_c645_48ab_a633_42fb816ad71c.slice/crio-e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6 WatchSource:0}: Error finding container e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6: Status 404 returned error can't find the container with id e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6
Oct 31 00:02:09.494785 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.494635    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:09.497428 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.497410    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14b5c6b_3af4_49b0_a5bb_930d814fa069.slice/crio-9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e WatchSource:0}: Error finding container 9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e: Status 404 returned error can't find the container with id 9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e
Oct 31 00:02:09.499503 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.499249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:09.502828 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.502801    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc469038_9017_4941_972e_0cb0bc3adf6c.slice/crio-d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3 WatchSource:0}: Error finding container d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3: Status 404 returned error can't find the container with id d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3
Oct 31 00:02:09.503709 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.503688    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffb287c_6e5e_4adb_9278_b26eec222aeb.slice/crio-ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f WatchSource:0}: Error finding container ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f: Status 404 returned error can't find the container with id ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f
Oct 31 00:02:09.508527 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:09.508509    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05191cb_2a10_4929_8000_b87e80b7070f.slice/crio-544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638 WatchSource:0}: Error finding container 544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638: Status 404 returned error can't find the container with id 544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638
Oct 31 00:02:09.779069 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.778992    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:09.779212 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.779081    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:09.779212 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.779202    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:09.779274 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.779250    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:10.779236011 +0000 UTC m=+4.111402523 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:09.779333 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.779300    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:09.779333 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.779324    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret podName:660c1701-2942-48bd-aee9-ff5341e6ffce nodeName:}" failed. No retries permitted until 2026-10-31 00:02:10.779317127 +0000 UTC m=+4.111483651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret") pod "global-pull-secret-syncer-gfzr6" (UID: "660c1701-2942-48bd-aee9-ff5341e6ffce") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:09.880143 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:09.880116    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:09.880310 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.880271    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:02:09.880310 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.880292    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:02:09.880310 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.880304    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9zp2t for pod openshift-network-diagnostics/network-check-target-s64vg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:09.880479 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:09.880359    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t podName:49220507-cecd-48d6-b19a-3f8479146036 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:10.880340354 +0000 UTC m=+4.212506873 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zp2t" (UniqueName: "kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t") pod "network-check-target-s64vg" (UID: "49220507-cecd-48d6-b19a-3f8479146036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:10.140502 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.140414    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f9k85" event={"ID":"d05191cb-2a10-4929-8000-b87e80b7070f","Type":"ContainerStarted","Data":"544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638"}
Oct 31 00:02:10.141748 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.141715    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4" event={"ID":"3ffb287c-6e5e-4adb-9278-b26eec222aeb","Type":"ContainerStarted","Data":"ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f"}
Oct 31 00:02:10.142919 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.142891    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-chjbs" event={"ID":"f14b5c6b-3af4-49b0-a5bb-930d814fa069","Type":"ContainerStarted","Data":"9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e"}
Oct 31 00:02:10.144409 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.144199    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bwpb" event={"ID":"19be463a-c645-48ab-a633-42fb816ad71c","Type":"ContainerStarted","Data":"e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6"}
Oct 31 00:02:10.145496 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.145472    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerStarted","Data":"aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62"}
Oct 31 00:02:10.146702 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.146680    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mnzps" event={"ID":"433c6858-00d2-47fc-8e98-9df265e3ce28","Type":"ContainerStarted","Data":"bb97eb920df58435d66cf84560fa183015f990a9e7f16f44132ff310ef754fe7"}
Oct 31 00:02:10.148013 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.147984    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cprhd" event={"ID":"a873a458-b1cc-4b84-9a61-a35b5c783868","Type":"ContainerStarted","Data":"562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef"}
Oct 31 00:02:10.149321 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.149297    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pqz2l" event={"ID":"bc469038-9017-4941-972e-0cb0bc3adf6c","Type":"ContainerStarted","Data":"d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3"}
Oct 31 00:02:10.150732 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.150706    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1"}
Oct 31 00:02:10.300634 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.300575    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:09 +0000 UTC" deadline="2028-07-20 18:25:58.480860808 +0000 UTC"
Oct 31 00:02:10.300634 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.300618    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15090h23m48.180248701s"
Oct 31 00:02:10.786335 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.786296    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:10.786505 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.786368    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:10.786574 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.786550    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:10.786627 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.786620    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret podName:660c1701-2942-48bd-aee9-ff5341e6ffce nodeName:}" failed. No retries permitted until 2026-10-31 00:02:12.786592595 +0000 UTC m=+6.118759128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret") pod "global-pull-secret-syncer-gfzr6" (UID: "660c1701-2942-48bd-aee9-ff5341e6ffce") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:10.786701 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.786686    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:10.786754 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.786724    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:12.786712028 +0000 UTC m=+6.118878553 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:10.887444 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:10.887350    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:10.887594 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.887581    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:02:10.887676 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.887602    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:02:10.887676 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.887614    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9zp2t for pod openshift-network-diagnostics/network-check-target-s64vg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:10.887676 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:10.887669    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t podName:49220507-cecd-48d6-b19a-3f8479146036 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:12.887652524 +0000 UTC m=+6.219819049 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zp2t" (UniqueName: "kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t") pod "network-check-target-s64vg" (UID: "49220507-cecd-48d6-b19a-3f8479146036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:11.141604 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:11.141578    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:11.142072 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:11.141714    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:11.142072 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:11.141794    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:11.142072 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:11.141861    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:11.142072 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:11.141916    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:11.142279 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:11.141982    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:11.301439 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:11.301358    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:09 +0000 UTC" deadline="2028-05-01 13:42:21.173957999 +0000 UTC"
Oct 31 00:02:11.301439 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:11.301436    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13165h40m9.872525942s"
Oct 31 00:02:12.185429 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.185246    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"3031cca7be5520d5a22eb64a065c3c72","Type":"ContainerStarted","Data":"7c699fcc7440f68d9a26851c3d71e085a03a8a388085fc79a8399e7d4f89d6d5"}
Oct 31 00:02:12.201558 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.201522    2574 generic.go:356] "Generic (PLEG): container finished" podID="f7aae722a65c2075f78dd611b7e7f739" containerID="ee0c8006630b804274d38b45146f13f80f48cb3bea2e94ca3c4fca404422c866" exitCode=0
Oct 31 00:02:12.201720 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.201574    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"f7aae722a65c2075f78dd611b7e7f739","Type":"ContainerDied","Data":"ee0c8006630b804274d38b45146f13f80f48cb3bea2e94ca3c4fca404422c866"}
Oct 31 00:02:12.427521 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.424565    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal" podStartSLOduration=4.424546528 podStartE2EDuration="4.424546528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:12.303581683 +0000 UTC m=+5.635748230" watchObservedRunningTime="2026-10-31 00:02:12.424546528 +0000 UTC m=+5.756713091"
Oct 31 00:02:12.804453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.804422    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:12.804596 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.804508    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:12.804668 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.804645    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:12.804729 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.804706    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:16.804686759 +0000 UTC m=+10.136853273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:12.804842 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.804772    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:12.804842 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.804806    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret podName:660c1701-2942-48bd-aee9-ff5341e6ffce nodeName:}" failed. No retries permitted until 2026-10-31 00:02:16.804796086 +0000 UTC m=+10.136962603 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret") pod "global-pull-secret-syncer-gfzr6" (UID: "660c1701-2942-48bd-aee9-ff5341e6ffce") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:12.906145 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:12.905868    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:12.906145 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.906011    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:02:12.906145 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.906037    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:02:12.906145 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.906053    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9zp2t for pod openshift-network-diagnostics/network-check-target-s64vg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:12.906145 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:12.906110    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t podName:49220507-cecd-48d6-b19a-3f8479146036 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:16.906092299 +0000 UTC m=+10.238258826 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zp2t" (UniqueName: "kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t") pod "network-check-target-s64vg" (UID: "49220507-cecd-48d6-b19a-3f8479146036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:13.135737 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.135668    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:13.135895 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:13.135776    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:13.135895 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.135823    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:13.135895 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:13.135874    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:13.136057 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.135910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:13.136057 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:13.135952    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:13.226869 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.226831    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"f7aae722a65c2075f78dd611b7e7f739","Type":"ContainerStarted","Data":"c248e2a1d2ce0bc4de9bfb788ed5cc55d89cc45563d726e7a7ad5fbee159ba22"}
Oct 31 00:02:13.235531 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.235502    2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="017a9e0577d43412556df290eb7675d974204fb2b4944acff685f861307075ac" exitCode=0
Oct 31 00:02:13.236454 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.236430    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerDied","Data":"017a9e0577d43412556df290eb7675d974204fb2b4944acff685f861307075ac"}
Oct 31 00:02:13.469018 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:13.468924    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" podStartSLOduration=5.468904936 podStartE2EDuration="5.468904936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:13.344930183 +0000 UTC m=+6.677096718" watchObservedRunningTime="2026-10-31 00:02:13.468904936 +0000 UTC m=+6.801071466"
Oct 31 00:02:14.245024 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:14.244940    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cprhd" event={"ID":"a873a458-b1cc-4b84-9a61-a35b5c783868","Type":"ContainerStarted","Data":"23e1246744359be6e970b0445baf269d6859ac5d302c75faf7a703a6f0a3081c"}
Oct 31 00:02:15.135966 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:15.135515    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:15.135966 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:15.135642    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:15.135966 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:15.135719    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:15.135966 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:15.135794    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:15.135966 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:15.135851    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:15.135966 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:15.135927    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:16.259276 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.259198    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f9k85" event={"ID":"d05191cb-2a10-4929-8000-b87e80b7070f","Type":"ContainerStarted","Data":"74d831d598b2a4b4782d1c8e5bfc3784f5739700c7bb8fadef20caed6abb5c06"}
Oct 31 00:02:16.264572 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.264540    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4" event={"ID":"3ffb287c-6e5e-4adb-9278-b26eec222aeb","Type":"ContainerStarted","Data":"22395e8c2e44be7bb7b8515ab771b86d2ca526aa878cff2908e430a94055769a"}
Oct 31 00:02:16.377158 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.377096    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cprhd" podStartSLOduration=5.094838601 podStartE2EDuration="9.377077045s" totalImagesPullingTime="4.282238444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:14.361514921 +0000 UTC m=+7.693681455" watchObservedRunningTime="2026-10-31 00:02:16.377077045 +0000 UTC m=+9.709243579"
Oct 31 00:02:16.789034 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.788997    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:16.790148 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.790126    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:16.842743 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.841980    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:16.842743 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.842045    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:16.842743 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.842185    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:16.842743 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.842254    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret podName:660c1701-2942-48bd-aee9-ff5341e6ffce nodeName:}" failed. No retries permitted until 2026-10-31 00:02:24.842233065 +0000 UTC m=+18.174399593 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret") pod "global-pull-secret-syncer-gfzr6" (UID: "660c1701-2942-48bd-aee9-ff5341e6ffce") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:16.842743 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.842677    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:16.842743 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.842712    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:24.842698484 +0000 UTC m=+18.174865008 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:16.916781 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.916715    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f9k85" podStartSLOduration=3.170318362 podStartE2EDuration="8.916687337s" totalImagesPullingTime="5.746368975s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:16.377936919 +0000 UTC m=+9.710103684" watchObservedRunningTime="2026-10-31 00:02:16.916687337 +0000 UTC m=+10.248853870"
Oct 31 00:02:16.942497 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:16.942455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:16.943036 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.942795    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:02:16.943036 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.942925    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:02:16.943036 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.942941    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9zp2t for pod openshift-network-diagnostics/network-check-target-s64vg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:16.943036 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:16.943008    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t podName:49220507-cecd-48d6-b19a-3f8479146036 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:24.94299139 +0000 UTC m=+18.275157916 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zp2t" (UniqueName: "kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t") pod "network-check-target-s64vg" (UID: "49220507-cecd-48d6-b19a-3f8479146036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:17.136441 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:17.136365    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:17.137165 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:17.136866    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:17.137165 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:17.136910    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:17.138316 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:17.137909    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:17.138316 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:17.137984    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:17.138591 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:17.137813    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:17.893564 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:17.893528    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:02:18.128499 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.128332    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:02:17.893549Z","UUID":"32b833bc-5eef-4bf1-825b-06ea1a75cd14","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:02:18.130123 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.130105    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:02:18.130228 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.130130    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:02:18.266730 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.266706    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:18.267204 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.267183    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:18.271887 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.271859    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4" event={"ID":"3ffb287c-6e5e-4adb-9278-b26eec222aeb","Type":"ContainerStarted","Data":"1bc9deb18b0451c8b8a576d2799d8a930f672056b9fda975a68cbe65c111d708"}
Oct 31 00:02:18.273864 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.273834    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-chjbs" event={"ID":"f14b5c6b-3af4-49b0-a5bb-930d814fa069","Type":"ContainerStarted","Data":"17df6711f1e798c5f53930a95e352e2cffe3e5ec1c01c65b8eb085ad41c7ef2f"}
Oct 31 00:02:18.276146 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.275914    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pqz2l" event={"ID":"bc469038-9017-4941-972e-0cb0bc3adf6c","Type":"ContainerStarted","Data":"f111da10a74f7b845593c36e5e27f2a567103263fcfa814d63ae61fa4573e5ff"}
Oct 31 00:02:18.523161 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.523097    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-chjbs" podStartSLOduration=3.224867645 podStartE2EDuration="11.523077897s" totalImagesPullingTime="8.298210252s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:18.52154614 +0000 UTC m=+11.853712674" watchObservedRunningTime="2026-10-31 00:02:18.523077897 +0000 UTC m=+11.855244431"
Oct 31 00:02:18.641206 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:18.641163    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-pqz2l" podStartSLOduration=3.545446604 podStartE2EDuration="11.641131806s" totalImagesPullingTime="8.095685202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:18.640737336 +0000 UTC m=+11.972903871" watchObservedRunningTime="2026-10-31 00:02:18.641131806 +0000 UTC m=+11.973298339"
Oct 31 00:02:19.133159 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:19.133088    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:19.133612 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:19.133229    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:19.133612 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:19.133287    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:19.133612 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:19.133332    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:19.133612 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:19.133407    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:19.133612 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:19.133467    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:19.282962 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:19.281568    2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="4e532db717b632feb6164721fe0bd3a3673344ce5026412da077aaeb08264c44" exitCode=0
Oct 31 00:02:19.282962 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:19.282592    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerDied","Data":"4e532db717b632feb6164721fe0bd3a3673344ce5026412da077aaeb08264c44"}
Oct 31 00:02:20.287210 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:20.287040    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4" event={"ID":"3ffb287c-6e5e-4adb-9278-b26eec222aeb","Type":"ContainerStarted","Data":"d741f6ce34954dcf3e15425cf45ed4fb4852f5a4e0a837ce54c4fa2f867c8f82"}
Oct 31 00:02:20.289431 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:20.289400    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9bwpb" event={"ID":"19be463a-c645-48ab-a633-42fb816ad71c","Type":"ContainerStarted","Data":"a08b1e32ba4045a4a81f4b4d0597f027b7e171426d0b2858aeda5031f6d7f455"}
Oct 31 00:02:20.291562 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:20.291536    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mnzps" event={"ID":"433c6858-00d2-47fc-8e98-9df265e3ce28","Type":"ContainerStarted","Data":"aa62afd075ee1ec4a063557ca5444a84aebe551c2b52337bcfac5787b9c414e7"}
Oct 31 00:02:20.409373 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:20.409324    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4" podStartSLOduration=3.797414301 podStartE2EDuration="13.409305368s" totalImagesPullingTime="9.611891067s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:20.408905603 +0000 UTC m=+13.741072136" watchObservedRunningTime="2026-10-31 00:02:20.409305368 +0000 UTC m=+13.741471932"
Oct 31 00:02:20.646014 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:20.645927    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mnzps" podStartSLOduration=3.268489941 podStartE2EDuration="13.645907863s" totalImagesPullingTime="10.377417922s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:20.530172496 +0000 UTC m=+13.862339028" watchObservedRunningTime="2026-10-31 00:02:20.645907863 +0000 UTC m=+13.978074390"
Oct 31 00:02:20.646151 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:20.646089    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9bwpb" podStartSLOduration=5.042690895 podStartE2EDuration="13.646082545s" totalImagesPullingTime="8.60339165s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:20.645974303 +0000 UTC m=+13.978140837" watchObservedRunningTime="2026-10-31 00:02:20.646082545 +0000 UTC m=+13.978249079"
Oct 31 00:02:21.133676 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:21.133249    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:21.133676 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:21.133293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:21.133676 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:21.133256    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:21.133676 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:21.133425    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:21.133676 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:21.133517    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:21.133676 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:21.133598    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:21.296694 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:21.296655    2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="e4ee70b6cb814d087991ee8729d166b11b991a4dd66787f28f695b4569f1810b" exitCode=0
Oct 31 00:02:21.297268 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:21.296754    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerDied","Data":"e4ee70b6cb814d087991ee8729d166b11b991a4dd66787f28f695b4569f1810b"}
Oct 31 00:02:23.133743 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:23.133705    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:23.134463 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:23.133717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:23.134463 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:23.133853    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:23.134463 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:23.133847    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:23.134463 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:23.133904    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:23.134463 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:23.133968    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:23.304012 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:23.303978    2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="8e7fbb10c7a14dbe06340268df6ee1a317b9dfe5ebd75b2a765f0d16cd487ce1" exitCode=0
Oct 31 00:02:23.304156 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:23.304027    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerDied","Data":"8e7fbb10c7a14dbe06340268df6ee1a317b9dfe5ebd75b2a765f0d16cd487ce1"}
Oct 31 00:02:24.309902 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.309671    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"0c63d8c1fe4131145d8d834d57591421888e17cfd4abf4f016e7e81688985683"}
Oct 31 00:02:24.310371 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.309925    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"e63dd2cf96758374f6cbedf53c8c587a4dc13a051b0102651946aabd84ab31ee"}
Oct 31 00:02:24.310371 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.309966    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"4fa4affd711aa10a7b4c3778676b259e15e6778883975a52cff6041e204bcb01"}
Oct 31 00:02:24.310371 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.309995    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"294c346c5616faa82f8ad3dc91ccb80fb2106b120d4d6da01d4e05551f0c27e9"}
Oct 31 00:02:24.310371 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.310028    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"70f28d8fe0c4861cfb8e3d53c9a8b346f4b647286766e0e3e66d57831c6ef648"}
Oct 31 00:02:24.310371 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.310058    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"cf94d84606d961fc8346d27bc1892b6b29f113ccf5d2466903c059773ed6d95d"}
Oct 31 00:02:24.912109 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.912076    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:24.912289 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:24.912121    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:24.912289 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:24.912224    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:24.912289 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:24.912229    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:24.912289 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:24.912286    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret podName:660c1701-2942-48bd-aee9-ff5341e6ffce nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.912267142 +0000 UTC m=+34.244433673 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret") pod "global-pull-secret-syncer-gfzr6" (UID: "660c1701-2942-48bd-aee9-ff5341e6ffce") : object "kube-system"/"original-pull-secret" not registered
Oct 31 00:02:24.912474 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:24.912305    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.912296054 +0000 UTC m=+34.244462571 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 31 00:02:25.013326 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:25.013290    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:25.013515 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.013487    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 31 00:02:25.013515 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.013510    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 31 00:02:25.013515 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.013520    2574 projected.go:196] Error preparing data for projected volume kube-api-access-9zp2t for pod openshift-network-diagnostics/network-check-target-s64vg: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:25.013717 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.013579    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t podName:49220507-cecd-48d6-b19a-3f8479146036 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:41.013561998 +0000 UTC m=+34.345728531 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9zp2t" (UniqueName: "kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t") pod "network-check-target-s64vg" (UID: "49220507-cecd-48d6-b19a-3f8479146036") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 31 00:02:25.133141 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:25.133107    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:25.133317 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:25.133180    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:25.133317 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:25.133199    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:25.133317 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.133282    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:25.133504 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.133455    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:25.133563 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:25.133540    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:26.316424 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:26.316229    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"db1e543d9aaa750a722c36fcb9f022c702189d5bd3130379708f82605ac54ea3"}
Oct 31 00:02:26.318638 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:26.318610    2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="7043b161f2be064558706c8c3eba856b6dfdfcf2f75f6c032672e7c2e629dcf2" exitCode=0
Oct 31 00:02:26.318757 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:26.318651    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerDied","Data":"7043b161f2be064558706c8c3eba856b6dfdfcf2f75f6c032672e7c2e629dcf2"}
Oct 31 00:02:27.133512 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:27.133478    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:27.133678 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:27.133565    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:27.133678 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:27.133607    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:27.133678 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:27.133644    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:27.133678 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:27.133678    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:27.133827 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:27.133725    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:27.323958 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:27.323930    2574 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="f39f539f9d9d8952b5abd7d2d0bc308366bd2a267f1082dbc5cf4125b1e84167" exitCode=0
Oct 31 00:02:27.324361 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:27.323969    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerDied","Data":"f39f539f9d9d8952b5abd7d2d0bc308366bd2a267f1082dbc5cf4125b1e84167"}
Oct 31 00:02:28.330171 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.330130    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" event={"ID":"f9cde72b-3aa1-4c45-95e1-72a55236dc1e","Type":"ContainerStarted","Data":"9d9fdb6c432afbf586f235eac846428a966c2d8b9c9ab60b97d556c36c5f9b66"}
Oct 31 00:02:28.330851 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.330562    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:28.330851 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.330645    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:28.335017 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.334984    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9b5br" event={"ID":"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2","Type":"ContainerStarted","Data":"cc5fb64733cb32862c474d37034177c5bbd25de921dfae69732c83aeee40343b"}
Oct 31 00:02:28.346274 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.346252    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:28.346753 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.346738    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:28.558917 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.558828    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh" podStartSLOduration=7.305684397 podStartE2EDuration="21.558816025s" totalImagesPullingTime="14.253131628s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:28.531845279 +0000 UTC m=+21.864011831" watchObservedRunningTime="2026-10-31 00:02:28.558816025 +0000 UTC m=+21.890982558"
Oct 31 00:02:28.655457 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:28.655402    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9b5br" podStartSLOduration=9.598260846 podStartE2EDuration="21.655369485s" totalImagesPullingTime="12.057108639s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:28.654908781 +0000 UTC m=+21.987075314" watchObservedRunningTime="2026-10-31 00:02:28.655369485 +0000 UTC m=+21.987536018"
Oct 31 00:02:29.133834 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:29.133749    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:29.133834 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:29.133787    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:29.134076 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:29.133845    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:29.134076 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:29.133898    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:29.134076 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:29.134002    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:29.134076 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:29.134056    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:29.337198 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:29.337173    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:02:30.339002 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:30.338970    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:02:31.077697 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:31.077491    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfzr6"]
Oct 31 00:02:31.077842 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:31.077780    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:31.077905 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:31.077889    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:31.080740 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:31.080717    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9cbv5"]
Oct 31 00:02:31.080866 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:31.080824    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:31.080942 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:31.080924    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:31.128229 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:31.128204    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s64vg"]
Oct 31 00:02:31.128324 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:31.128283    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:31.128371 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:31.128342    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:32.132895 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:32.132867    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:32.133344 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:32.132970    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:33.133591 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:33.133563    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:33.133941 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:33.133705    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:33.133941 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:33.133739    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:33.133941 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:33.133815    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:34.133649 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:34.133621    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:34.134016 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:34.133717    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:35.133511 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:35.133445    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:35.133663 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:35.133540    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-s64vg" podUID="49220507-cecd-48d6-b19a-3f8479146036"
Oct 31 00:02:35.133663 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:35.133627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:35.134213 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:35.134185    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9cbv5" podUID="1a6de299-a94e-4591-acac-b5018a133c65"
Oct 31 00:02:35.929863 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:35.929799    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:35.930030 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:35.930018    2574 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:02:35.943461 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:35.943433    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:36.133479 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.133452    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:36.133602 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:36.133546    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfzr6" podUID="660c1701-2942-48bd-aee9-ff5341e6ffce"
Oct 31 00:02:36.422253 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.422229    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeReady"
Oct 31 00:02:36.422611 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.422346    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 31 00:02:36.549985 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.549918    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hlhj6"]
Oct 31 00:02:36.563215 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.563194    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gjbb9"]
Oct 31 00:02:36.563319 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.563305    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.575635 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.575617    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.616115 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.616096    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlhj6"]
Oct 31 00:02:36.619617 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.619599    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:02:36.619828 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.619814    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtt85\""
Oct 31 00:02:36.623569 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.623541    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjbb9"]
Oct 31 00:02:36.623659 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.623627    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:02:36.700049 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.700018    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.700170 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.700065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e4fdae-7252-4a99-9f43-3939b224757c-tmp-dir\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.700170 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.700087    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7lt8g\" (UniqueName: \"kubernetes.io/projected/89ac4cee-c777-4655-9bcc-aa09aa136ab5-kube-api-access-7lt8g\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.700243 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.700185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89ac4cee-c777-4655-9bcc-aa09aa136ab5-cert\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.700243 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.700213    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mddmq\" (UniqueName: \"kubernetes.io/projected/b4e4fdae-7252-4a99-9f43-3939b224757c-kube-api-access-mddmq\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.700303 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.700243    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4fdae-7252-4a99-9f43-3939b224757c-config-volume\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.719769 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.719746    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hplth\""
Oct 31 00:02:36.719769 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.719749    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:02:36.723418 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.723404    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:02:36.723418 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.723415    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:02:36.800830 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.800746    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lt8g\" (UniqueName: \"kubernetes.io/projected/89ac4cee-c777-4655-9bcc-aa09aa136ab5-kube-api-access-7lt8g\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.800830 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.800788    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89ac4cee-c777-4655-9bcc-aa09aa136ab5-cert\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.801081 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.800855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddmq\" (UniqueName: \"kubernetes.io/projected/b4e4fdae-7252-4a99-9f43-3939b224757c-kube-api-access-mddmq\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.801081 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.800919    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4fdae-7252-4a99-9f43-3939b224757c-config-volume\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.801081 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.800942    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.801081 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.800994    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e4fdae-7252-4a99-9f43-3939b224757c-tmp-dir\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.801303 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:36.801092    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:36.801303 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:36.801172    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:37.301151061 +0000 UTC m=+30.633317571 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:36.801435 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.801305    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e4fdae-7252-4a99-9f43-3939b224757c-tmp-dir\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.801517 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.801500    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4fdae-7252-4a99-9f43-3939b224757c-config-volume\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.804811 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.804791    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89ac4cee-c777-4655-9bcc-aa09aa136ab5-cert\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.865353 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.865328    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddmq\" (UniqueName: \"kubernetes.io/projected/b4e4fdae-7252-4a99-9f43-3939b224757c-kube-api-access-mddmq\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:36.865555 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.865540    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lt8g\" (UniqueName: \"kubernetes.io/projected/89ac4cee-c777-4655-9bcc-aa09aa136ab5-kube-api-access-7lt8g\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:36.906792 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:36.906612    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:37.134092 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.133859    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:37.134252 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.133861    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:37.188483 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.188459    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:02:37.188623 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.188485    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:02:37.188679 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.188625    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:02:37.188679 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.188660    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\""
Oct 31 00:02:37.188778 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.188719    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\""
Oct 31 00:02:37.305694 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.305666    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:37.305839 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:37.305799    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:37.305882 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:37.305868    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:38.30585335 +0000 UTC m=+31.638019876 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:37.326362 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.326334    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gjbb9"]
Oct 31 00:02:37.329928 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:37.329908    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89ac4cee_c777_4655_9bcc_aa09aa136ab5.slice/crio-74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b WatchSource:0}: Error finding container 74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b: Status 404 returned error can't find the container with id 74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b
Oct 31 00:02:37.354130 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:37.354106    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjbb9" event={"ID":"89ac4cee-c777-4655-9bcc-aa09aa136ab5","Type":"ContainerStarted","Data":"74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b"}
Oct 31 00:02:38.133018 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:38.132986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:38.188926 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:38.188894    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\""
Oct 31 00:02:38.189114 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:38.188971    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:02:38.313234 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:38.313205    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:38.313416 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:38.313340    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:38.313505 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:38.313438    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:40.313419201 +0000 UTC m=+33.645585728 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:40.328992 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:40.328917    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:40.329570 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:40.329053    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:40.329570 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:40.329128    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.329106428 +0000 UTC m=+37.661272962 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:40.362099 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:40.362072    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gjbb9" event={"ID":"89ac4cee-c777-4655-9bcc-aa09aa136ab5","Type":"ContainerStarted","Data":"e27ee54e98251db5cfce58ef2265e34f1405a7b3c6dd893371fcdabce8fb5ab9"}
Oct 31 00:02:40.479790 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:40.479743    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gjbb9" podStartSLOduration=1.754838933 podStartE2EDuration="4.479729388s" totalImagesPullingTime="2.724890455s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:40.478868855 +0000 UTC m=+33.811035387" watchObservedRunningTime="2026-10-31 00:02:40.479729388 +0000 UTC m=+33.811895921"
Oct 31 00:02:40.932693 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:40.932667    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:40.932821 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:40.932741    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:40.932871 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:40.932857    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:02:40.932933 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:40.932924    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:12.932907809 +0000 UTC m=+66.265074342 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : secret "metrics-daemon-secret" not found
Oct 31 00:02:40.935921 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:40.935904    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:41.033476 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:41.033453    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:41.035873 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:41.035855    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:41.062280 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:41.062259    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:41.161332 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:41.161300    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:41.429786 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:41.429764    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-s64vg"]
Oct 31 00:02:41.431348 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:41.431319    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49220507_cecd_48d6_b19a_3f8479146036.slice/crio-e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63 WatchSource:0}: Error finding container e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63: Status 404 returned error can't find the container with id e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63
Oct 31 00:02:41.531750 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:41.531689    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfzr6"]
Oct 31 00:02:41.534314 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:41.534286    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660c1701_2942_48bd_aee9_ff5341e6ffce.slice/crio-9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304 WatchSource:0}: Error finding container 9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304: Status 404 returned error can't find the container with id 9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304
Oct 31 00:02:42.368305 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:42.368264    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gfzr6" event={"ID":"660c1701-2942-48bd-aee9-ff5341e6ffce","Type":"ContainerStarted","Data":"9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304"}
Oct 31 00:02:42.369441 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:42.369420    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s64vg" event={"ID":"49220507-cecd-48d6-b19a-3f8479146036","Type":"ContainerStarted","Data":"e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63"}
Oct 31 00:02:43.724919 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.724885    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mvljb"]
Oct 31 00:02:43.727813 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.726939    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.780701 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.780673    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:02:43.780856 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.780779    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:02:43.831448 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.831391    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:02:43.831448 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.831404    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:02:43.831448 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.831431    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4gwp\""
Oct 31 00:02:43.832730 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.832696    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvljb"]
Oct 31 00:02:43.855141 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.855103    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-crio-socket\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.855248 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.855160    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.855304 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.855274    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-data-volume\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.855343 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.855313    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phbmw\" (UniqueName: \"kubernetes.io/projected/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-api-access-phbmw\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.855397 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.855371    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.955971 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.955938    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbmw\" (UniqueName: \"kubernetes.io/projected/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-api-access-phbmw\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.956131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.955989    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.956131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.956049    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-crio-socket\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.956131 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.956087    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.956235 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.956131    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-data-volume\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.956517 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.956494    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-data-volume\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.956891 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:43.956875    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:02:43.956954 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:43.956944    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.456925862 +0000 UTC m=+37.789092392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : configmap "kube-rbac-proxy" not found
Oct 31 00:02:43.957308 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.957287    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-crio-socket\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:43.957412 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:43.957359    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:43.957495 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:43.957483    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:44.457466562 +0000 UTC m=+37.789633091 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:43.976109 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.976051    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-4xtcf-rd6pb"]
Oct 31 00:02:43.979329 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:43.978752    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.022824 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.022799    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbmw\" (UniqueName: \"kubernetes.io/projected/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-api-access-phbmw\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:44.056445 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.056406    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.056593 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.056471    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.056593 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.056542    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.056593 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.056568    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.083636 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.083597    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkxnt\""
Oct 31 00:02:44.084914 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.084884    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:02:44.117292 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.117265    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:02:44.118067 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.118049    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4xtcf-rd6pb"]
Oct 31 00:02:44.157286 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.157257    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.157453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.157314    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.157453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.157397    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.157453 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.157426    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.157774 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.157748    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.158367 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.158343    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.158534 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.158514    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.220964 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.220931    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.291938 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.291864    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:44.359987 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.359951    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:44.360116 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:44.360098    2574 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:44.360183 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:44.360168    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:52.360149336 +0000 UTC m=+45.692315849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:44.461911 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.461263    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:44.461911 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.461347    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:44.461911 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:44.461497    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:44.461911 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:44.461551    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:45.461538983 +0000 UTC m=+38.793705497 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:44.461911 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:44.461589    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:02:44.461911 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:44.461610    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:45.461603041 +0000 UTC m=+38.793769552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : configmap "kube-rbac-proxy" not found
Oct 31 00:02:44.693785 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.693743    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-4xtcf-rd6pb"]
Oct 31 00:02:44.695525 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.695505    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"]
Oct 31 00:02:44.697578 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:44.697535    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f241cd5_d596_4fb1_a270_39d5f3a83727.slice/crio-07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871 WatchSource:0}: Error finding container 07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871: Status 404 returned error can't find the container with id 07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871
Oct 31 00:02:44.697879 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.697857    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:44.753881 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.753854    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zm4vk\""
Oct 31 00:02:44.754969 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.754941    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:02:44.757830 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.757809    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:02:44.759289 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.759266    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"]
Oct 31 00:02:44.868216 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.868134    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qvx\" (UniqueName: \"kubernetes.io/projected/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae-kube-api-access-z2qvx\") pod \"migrator-66df5d496b-db4f4\" (UID: \"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:44.969465 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:44.969321    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qvx\" (UniqueName: \"kubernetes.io/projected/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae-kube-api-access-z2qvx\") pod \"migrator-66df5d496b-db4f4\" (UID: \"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:45.031474 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.031441    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qvx\" (UniqueName: \"kubernetes.io/projected/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae-kube-api-access-z2qvx\") pod \"migrator-66df5d496b-db4f4\" (UID: \"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:45.316484 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.316409    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:45.378933 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.378897    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb" event={"ID":"6f241cd5-d596-4fb1-a270-39d5f3a83727","Type":"ContainerStarted","Data":"07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871"}
Oct 31 00:02:45.380500 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.380477    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-s64vg" event={"ID":"49220507-cecd-48d6-b19a-3f8479146036","Type":"ContainerStarted","Data":"862fe3c91f5680f22de03025f8c3d449983035140a0cc98ff505b55ddd283f66"}
Oct 31 00:02:45.381196 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.381179    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:45.473458 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.473424    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:45.473670 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.473496    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:45.473852 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:45.473829    2574 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:45.473948 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:45.473908    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:47.473887136 +0000 UTC m=+40.806053656 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:45.473948 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:45.473831    2574 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:02:45.474062 ip-10-0-137-148 kubenswrapper[2574]: E1031 00:02:45.473995    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:47.47397595 +0000 UTC m=+40.806142481 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : configmap "kube-rbac-proxy" not found
Oct 31 00:02:45.498157 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.498116    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-s64vg" podStartSLOduration=35.001811324 podStartE2EDuration="38.498104288s" totalImagesPullingTime="3.496292964s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:02:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:02:45.497283915 +0000 UTC m=+38.829450447" watchObservedRunningTime="2026-10-31 00:02:45.498104288 +0000 UTC m=+38.830270820"
Oct 31 00:02:45.663918 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.663851    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-bkzrc"]
Oct 31 00:02:45.666563 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.666543    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.689439 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.689414    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"]
Oct 31 00:02:45.692448 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:45.692414    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod868e0cf5_15d2_4a00_9fe6_01ff90ac0aae.slice/crio-fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8 WatchSource:0}: Error finding container fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8: Status 404 returned error can't find the container with id fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8
Oct 31 00:02:45.723611 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.723591    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hg87m\""
Oct 31 00:02:45.723702 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.723690    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:02:45.724192 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.724174    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:02:45.724263 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.724243    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:02:45.724320 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.724310    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:02:45.724397 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.724358    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-bkzrc"]
Oct 31 00:02:45.775104 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.775074    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhzm8\" (UniqueName: \"kubernetes.io/projected/14418f52-35b7-4c4e-9593-cc99bedcb693-kube-api-access-fhzm8\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.775547 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.775145    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-key\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.775547 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.775292    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-cabundle\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.876940 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.876446    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-key\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.876940 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.876540    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-cabundle\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.876940 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.876611    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzm8\" (UniqueName: \"kubernetes.io/projected/14418f52-35b7-4c4e-9593-cc99bedcb693-kube-api-access-fhzm8\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.877269 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.877245    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-cabundle\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.879612 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.879592    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-key\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.939733 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.939707    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzm8\" (UniqueName: \"kubernetes.io/projected/14418f52-35b7-4c4e-9593-cc99bedcb693-kube-api-access-fhzm8\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:45.977711 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:45.977682    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:46.357154 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:46.357068    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-bkzrc"]
Oct 31 00:02:46.361456 ip-10-0-137-148 kubenswrapper[2574]: W1031 00:02:46.361425    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14418f52_35b7_4c4e_9593_cc99bedcb693.slice/crio-85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b WatchSource:0}: Error finding container 85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b: Status 404 returned error can't find the container with id 85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b
Oct 31 00:02:46.383848 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:46.383821    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4" event={"ID":"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae","Type":"ContainerStarted","Data":"fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8"}
Oct 31 00:02:46.385164 ip-10-0-137-148 kubenswrapper[2574]: I1031 00:02:46.385137    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc" event={"ID":"14418f52-35b7-4c4e-9593-cc99bedcb693","Type":"ContainerStarted","Data":"85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b"}
Oct 31 00:02:46.919117 ip-10-0-137-148 systemd[1]: Stopping Kubernetes Kubelet...
Oct 31 00:02:46.929875 ip-10-0-137-148 systemd[1]: kubelet.service: Deactivated successfully.
Oct 31 00:02:46.930100 ip-10-0-137-148 systemd[1]: Stopped Kubernetes Kubelet.
Oct 31 00:02:46.930160 ip-10-0-137-148 systemd[1]: kubelet.service: Consumed 2.220s CPU time.
Oct 31 00:02:46.939969 ip-10-0-137-148 systemd[1]: Starting Kubernetes Kubelet...
Oct 31 00:02:47.019234 ip-10-0-137-148 kubenswrapper[4478]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:02:47.019234 ip-10-0-137-148 kubenswrapper[4478]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 31 00:02:47.019234 ip-10-0-137-148 kubenswrapper[4478]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:02:47.019234 ip-10-0-137-148 kubenswrapper[4478]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 31 00:02:47.021547 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021469    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.021547 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021543    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.021547 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021548    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.021547 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021552    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021556    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021561    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021568    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021574    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021578    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021582    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021586    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021590    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021594    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021598    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021612    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021617    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021620    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021624    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021628    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021632    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021636    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021640    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.021809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021644    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021647    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021652    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021656    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021660    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021664    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021667    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021671    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021675    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021679    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021684    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021688    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021692    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021696    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021700    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021706    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021710    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021715    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021719    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021723    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.022634 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021728    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021732    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021736    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021741    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021745    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021751    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021755    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021759    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021764    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021768    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021772    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021776    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021780    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021784    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021789    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021793    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021797    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021802    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021806    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021810    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.023559 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021814    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021819    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021824    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021828    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021833    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021837    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021841    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021846    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021850    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021854    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021858    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021862    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021867    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021871    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021875    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021880    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021884    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021888    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021893    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021897    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.024443 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021903    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021907    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021911    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021916    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021920    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021924    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021929    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021933    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021937    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021942    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021946    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021950    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021954    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021959    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021963    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021967    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021973    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021979    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.025306 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021983    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021988    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021993    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.021997    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.022004    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.022008    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023300    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023310    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023315    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023320    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023325    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023329    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023334    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023361    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023368    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023373    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023396    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023402    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023407    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.026107 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023411    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023415    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023419    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023423    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023428    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023432    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023436    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023440    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023444    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023448    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023452    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023457    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023461    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023465    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023470    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023474    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023478    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023483    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023487    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023491    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.027036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023495    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023502    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023507    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023511    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023515    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023519    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023523    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023528    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023533    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023537    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023542    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023546    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023550    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023555    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023560    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023566    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023570    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023575    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023579    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.027886 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023583    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023587    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023591    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023595    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023599    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023603    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023607    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023612    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023617    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023621    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023625    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023630    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023634    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023638    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023645    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023649    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023653    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023657    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023662    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.028374 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023666    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023670    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023675    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023679    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023683    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023687    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023695    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023701    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023706    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023712    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023716    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023720    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023725    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023730    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023734    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023739    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023744    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023748    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023752    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.028918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023756    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023761    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023765    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023769    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023773    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023778    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023782    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023786    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023792    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023796    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023800    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023806    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023810    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023815    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023819    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.023823    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024637    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024647    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024651    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024656    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.029452 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024660    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024665    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024669    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024673    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024682    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024687    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024691    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024695    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024699    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024703    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024707    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024712    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024715    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024719    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024724    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024728    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024732    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024736    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024740    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024745    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.030013 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024750    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024755    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024759    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024763    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024767    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024772    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024776    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024780    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024785    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024789    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024793    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024797    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024801    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024807    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024813    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024818    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024825    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024830    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024834    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.030544 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024838    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024844    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024848    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024853    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024857    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024862    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024866    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024870    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024874    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024878    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024882    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024886    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024890    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024896    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024900    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024904    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024909    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024913    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024917    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024921    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.031033 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024925    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024929    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024934    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024938    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024943    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024947    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024951    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024955    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024959    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024968    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024973    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024978    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024983    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024987    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024992    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.024996    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025000    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025005    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025009    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.031541 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025013    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025017    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025021    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025024    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025029    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025033    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025039    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025043    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025047    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025052    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025056    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025060    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025064    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025068    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025072    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025077    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025082    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025086    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025090    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025094    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.032040 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025098    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025102    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025107    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.025112    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025253    4478 flags.go:64] FLAG: --address="0.0.0.0"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025270    4478 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025281    4478 flags.go:64] FLAG: --anonymous-auth="true"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025287    4478 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025294    4478 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025300    4478 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025306    4478 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025313    4478 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025317    4478 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025322    4478 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025327    4478 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025333    4478 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025338    4478 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025343    4478 flags.go:64] FLAG: --cgroup-root=""
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025348    4478 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025352    4478 flags.go:64] FLAG: --client-ca-file=""
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025357    4478 flags.go:64] FLAG: --cloud-provider="external"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025361    4478 flags.go:64] FLAG: --cluster-dns="[]"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025367    4478 flags.go:64] FLAG: --cluster-domain=""
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025371    4478 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 31 00:02:47.032582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025395    4478 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025401    4478 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025406    4478 flags.go:64] FLAG: --container-log-max-files="5"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025413    4478 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025417    4478 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025423    4478 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025429    4478 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025433    4478 flags.go:64] FLAG: --contention-profiling="false"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025438    4478 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025442    4478 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025450    4478 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025455    4478 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025461    4478 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025466    4478 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025471    4478 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025476    4478 flags.go:64] FLAG: --enable-load-reader="false"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025480    4478 flags.go:64] FLAG: --enable-server="true"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025485    4478 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025491    4478 flags.go:64] FLAG: --event-burst="100"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025497    4478 flags.go:64] FLAG: --event-qps="50"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025502    4478 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025507    4478 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025512    4478 flags.go:64] FLAG: --eviction-hard=""
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025518    4478 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 31 00:02:47.033241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025523    4478 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025528    4478 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025533    4478 flags.go:64] FLAG: --eviction-soft=""
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025537    4478 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025542    4478 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025546    4478 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025551    4478 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025555    4478 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025560    4478 flags.go:64] FLAG: --fail-swap-on="true"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025565    4478 flags.go:64] FLAG: --feature-gates=""
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025571    4478 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025576    4478 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025581    4478 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025586    4478 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025592    4478 flags.go:64] FLAG: --healthz-port="10248"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025597    4478 flags.go:64] FLAG: --help="false"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025602    4478 flags.go:64] FLAG: --hostname-override="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025608    4478 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025614    4478 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025619    4478 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025625    4478 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025631    4478 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025635    4478 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 31 00:02:47.034308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025640    4478 flags.go:64] FLAG: --image-service-endpoint=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025644    4478 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025649    4478 flags.go:64] FLAG: --kube-api-burst="100"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025654    4478 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025659    4478 flags.go:64] FLAG: --kube-api-qps="50"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025663    4478 flags.go:64] FLAG: --kube-reserved=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025668    4478 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025672    4478 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025678    4478 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025682    4478 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025687    4478 flags.go:64] FLAG: --lock-file=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025691    4478 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025695    4478 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025700    4478 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025709    4478 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025713    4478 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025718    4478 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025723    4478 flags.go:64] FLAG: --logging-format="text"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025727    4478 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025732    4478 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025737    4478 flags.go:64] FLAG: --manifest-url=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025741    4478 flags.go:64] FLAG: --manifest-url-header=""
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025749    4478 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025753    4478 flags.go:64] FLAG: --max-open-files="1000000"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025766    4478 flags.go:64] FLAG: --max-pods="110"
Oct 31 00:02:47.035083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025770    4478 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025775    4478 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025783    4478 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025787    4478 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025792    4478 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025796    4478 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025801    4478 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025811    4478 flags.go:64] FLAG: --node-status-max-images="50"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025815    4478 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025820    4478 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025825    4478 flags.go:64] FLAG: --pod-cidr=""
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025829    4478 flags.go:64] FLAG: --pod-manifest-path=""
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025834    4478 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025838    4478 flags.go:64] FLAG: --pods-per-core="0"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025843    4478 flags.go:64] FLAG: --port="10250"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025847    4478 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025852    4478 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-094cae1e0a98e38aa"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025857    4478 flags.go:64] FLAG: --qos-reserved=""
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025861    4478 flags.go:64] FLAG: --read-only-port="10255"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025866    4478 flags.go:64] FLAG: --register-node="true"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025870    4478 flags.go:64] FLAG: --register-with-taints=""
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025877    4478 flags.go:64] FLAG: --registry-burst="10"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025881    4478 flags.go:64] FLAG: --registry-qps="5"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025885    4478 flags.go:64] FLAG: --reserved-cpus=""
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025892    4478 flags.go:64] FLAG: --reserved-memory=""
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025898    4478 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 31 00:02:47.035873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025903    4478 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025907    4478 flags.go:64] FLAG: --rotate-certificates="false"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025912    4478 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025917    4478 flags.go:64] FLAG: --runonce="false"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025922    4478 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025926    4478 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025932    4478 flags.go:64] FLAG: --seccomp-default="false"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025937    4478 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025943    4478 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025948    4478 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025953    4478 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025957    4478 flags.go:64] FLAG: --storage-driver-password="root"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025962    4478 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025967    4478 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025971    4478 flags.go:64] FLAG: --storage-driver-user="root"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025976    4478 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025980    4478 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025985    4478 flags.go:64] FLAG: --system-cgroups=""
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025990    4478 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.025998    4478 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026003    4478 flags.go:64] FLAG: --tls-cert-file=""
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026007    4478 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026012    4478 flags.go:64] FLAG: --tls-min-version=""
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026017    4478 flags.go:64] FLAG: --tls-private-key-file=""
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026021    4478 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 31 00:02:47.036628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026025    4478 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026030    4478 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026035    4478 flags.go:64] FLAG: --v="2"
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026041    4478 flags.go:64] FLAG: --version="false"
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026048    4478 flags.go:64] FLAG: --vmodule=""
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026056    4478 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026061    4478 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026273    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026282    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026288    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026293    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026298    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026302    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026306    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026312    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026318    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026322    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026326    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026330    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026334    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026339    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.037331 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026344    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026348    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026352    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026356    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026360    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026364    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026369    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026373    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026395    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026400    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026404    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026407    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026411    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026416    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026420    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026424    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026430    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026434    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026439    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026443    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.038038 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026447    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026454    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026461    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026467    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026472    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026477    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026483    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026487    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026492    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026496    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026500    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026504    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026508    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026513    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026517    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026522    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026526    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026530    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026535    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.038644 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026539    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026543    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026547    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026551    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026556    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026560    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026564    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026568    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026573    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026578    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026583    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026587    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026592    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026596    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026600    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026604    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026608    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026613    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026617    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.039170 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026622    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026627    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026631    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026636    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026640    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026644    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026649    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026653    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026657    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026661    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026665    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026670    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026674    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026678    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026682    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026687    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026691    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026696    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026700    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026704    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.039750 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026708    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026719    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026725    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026730    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026734    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026738    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026742    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026746    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026751    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026755    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026759    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026764    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026769    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026773    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.026818    4478 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026871    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.040308 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026878    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026883    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026888    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026892    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026896    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026900    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026905    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026909    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026913    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026917    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026921    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026926    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026930    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026934    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026938    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026943    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026947    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026951    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026957    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026962    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.040892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026966    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026970    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026974    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026978    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026982    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026986    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026990    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026995    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.026999    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027005    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027009    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027015    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027019    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027023    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027027    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027031    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027035    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027039    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027043    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.041480 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027048    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027052    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027056    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027060    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027064    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027068    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027074    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027080    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027086    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027091    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027095    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027101    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027105    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027109    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027114    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027118    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027122    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027126    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027130    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.041998 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027135    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027141    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027146    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027150    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027154    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027158    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027163    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027168    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027172    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027176    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027180    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027184    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027188    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027192    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027196    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027200    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027205    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027209    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027213    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.042632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027217    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027222    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027227    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027233    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027239    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027247    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027252    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027257    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027261    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027265    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027269    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027274    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027278    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027282    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027286    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027290    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027294    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027299    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027303    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.043182 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027307    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027312    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027316    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027321    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027326    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027330    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027335    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027339    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.027343    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.033402    4478 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.033424    4478 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.033450    4478 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.033457    4478 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033833    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033843    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033847    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033852    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033856    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033860    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033864    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.043781 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033869    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033879    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033884    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033888    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033893    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033897    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033901    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033905    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033910    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033914    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033918    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033922    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033926    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033931    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033940    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033944    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033948    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033952    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033956    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033960    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.044483 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033965    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.033969    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034006    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034012    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034016    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034020    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034030    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034034    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034039    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034043    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034047    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034051    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034055    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034059    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034063    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034067    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034071    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034074    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034077    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034124    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.045024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034142    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034148    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034155    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034160    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034165    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034170    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034175    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034180    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034226    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034240    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034247    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034252    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034257    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034262    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034267    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034271    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034277    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034282    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034287    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034291    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.045645 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034301    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034305    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034309    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034313    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034318    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034322    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034327    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034331    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034335    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034339    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034343    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034348    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034352    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034367    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034372    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034394    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034399    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034403    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034407    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.046225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034410    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034415    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034437    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034444    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034449    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034453    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034458    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034462    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034469    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034501    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034650    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034663    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034668    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034672    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034677    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034682    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034686    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034690    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.046822 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034695    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.034699    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.034760    4478 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035030    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035040    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035045    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035049    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035052    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035056    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035058    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035061    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035064    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035066    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035069    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035072    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.047314 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035075    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035078    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035082    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035086    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035091    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035096    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035101    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035105    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035111    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035115    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035119    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035124    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035128    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035132    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035137    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035141    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035145    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035149    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035153    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035157    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.047782 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035161    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035166    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035170    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035176    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035180    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035185    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035189    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035193    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035196    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035199    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035201    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035204    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035206    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035209    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035212    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035214    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035217    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035220    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035222    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035225    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.048335 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035228    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035230    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035233    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035235    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035238    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035241    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035243    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035246    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035249    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035251    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035254    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035256    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035259    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035262    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035264    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035267    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035269    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035272    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035274    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035278    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.048937 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035283    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035287    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035291    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035296    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035300    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035303    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035306    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035308    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035311    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035314    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035316    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035319    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035321    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035324    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035326    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035329    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035331    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035334    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035338    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035342    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.049531 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035345    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035349    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035352    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035354    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035358    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035360    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035363    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035366    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035369    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035373    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035398    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035402    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035406    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035409    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.035457    4478 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 31 00:02:47.050160 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035506    4478 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035511    4478 feature_gate.go:386] unrecognized feature gate: Example
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035515    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035520    4478 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035525    4478 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035529    4478 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035534    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035536    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035539    4478 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035542    4478 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035545    4478 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035547    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035550    4478 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035552    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035555    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035557    4478 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035560    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035564    4478 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035568    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 31 00:02:47.050668 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035573    4478 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035576    4478 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035580    4478 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035582    4478 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035585    4478 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035587    4478 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035590    4478 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035593    4478 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035596    4478 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035598    4478 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035601    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035604    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035607    4478 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035609    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035612    4478 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035615    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035617    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035620    4478 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035624    4478 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035629    4478 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 31 00:02:47.051252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035633    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035636    4478 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035640    4478 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035644    4478 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035647    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035651    4478 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035656    4478 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035661    4478 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035667    4478 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035672    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035675    4478 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035678    4478 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035681    4478 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035685    4478 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035689    4478 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035693    4478 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035698    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035702    4478 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 31 00:02:47.051894 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035706    4478 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035710    4478 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035714    4478 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035718    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035722    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035727    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035733    4478 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035737    4478 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035741    4478 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035746    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035751    4478 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035755    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035761    4478 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035765    4478 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035769    4478 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035772    4478 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035774    4478 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035777    4478 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035779    4478 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035781    4478 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 31 00:02:47.052412 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035784    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035789    4478 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035793    4478 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035798    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035802    4478 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035806    4478 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035809    4478 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035812    4478 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035815    4478 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035817    4478 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035820    4478 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035823    4478 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035827    4478 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035831    4478 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035835    4478 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035839    4478 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035843    4478 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035847    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035851    4478 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035854    4478 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 31 00:02:47.052963 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035857    4478 feature_gate.go:386] unrecognized feature gate: Example2
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035860    4478 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035862    4478 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035865    4478 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035867    4478 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035870    4478 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035872    4478 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035875    4478 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:47.035878    4478 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.035939    4478 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037071    4478 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037139    4478 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037329    4478 server.go:1014] "Starting client certificate rotation"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037520    4478 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037569    4478 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:57:07 +0000 UTC" deadline="2028-07-20 20:08:12.488111772 +0000 UTC"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037640    4478 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15092h5m25.45047473s"
Oct 31 00:02:47.053579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.037977    4478 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:02:47.054032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.039235    4478 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 31 00:02:47.054032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.041044    4478 log.go:25] "Validated CRI v1 runtime API"
Oct 31 00:02:47.054032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.042848    4478 log.go:25] "Validated CRI v1 image API"
Oct 31 00:02:47.054032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.043676    4478 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 31 00:02:47.054032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.045396    4478 fs.go:135] Filesystem UUIDs: map[34186ae9-c764-48a7-8bf7-0b175261029f:/dev/nvme0n1p3 48116720-39b3-4a68-8baa-c9ff8aab0ded:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Oct 31 00:02:47.054232 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.045419    4478 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3/userdata/shm major:0 minor:91 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6/userdata/shm major:0 minor:79 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8/userdata/shm major:0 minor:235 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14418f52-35b7-4c4e-9593-cc99bedcb693/volumes/kubernetes.io~projected/kube-api-access-fhzm8:{mountpoint:/var/lib/kubelet/pods/14418f52-35b7-4c4e-9593-cc99bedcb693/volumes/kubernetes.io~projected/kube-api-access-fhzm8 major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14418f52-35b7-4c4e-9593-cc99bedcb693/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/14418f52-35b7-4c4e-9593-cc99bedcb693/volumes/kubernetes.io~secret/signing-key major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19be463a-c645-48ab-a633-42fb816ad71c/volumes/kubernetes.io~projected/kube-api-access-gp4rs:{mountpoint:/var/lib/kubelet/pods/19be463a-c645-48ab-a633-42fb816ad71c/volumes/kubernetes.io~projected/kube-api-access-gp4rs major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1a6de299-a94e-4591-acac-b5018a133c65/volumes/kubernetes.io~projected/kube-api-access-lvb88:{mountpoint:/var/lib/kubelet/pods/1a6de299-a94e-4591-acac-b5018a133c65/volumes/kubernetes.io~projected/kube-api-access-lvb88 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ffb287c-6e5e-4adb-9278-b26eec222aeb/volumes/kubernetes.io~projected/kube-api-access-fd8tq:{mountpoint:/var/lib/kubelet/pods/3ffb287c-6e5e-4adb-9278-b26eec222aeb/volumes/kubernetes.io~projected/kube-api-access-fd8tq major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~projected/kube-api-access-s5z6b:{mountpoint:/var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~projected/kube-api-access-s5z6b major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/49220507-cecd-48d6-b19a-3f8479146036/volumes/kubernetes.io~projected/kube-api-access-9zp2t:{mountpoint:/var/lib/kubelet/pods/49220507-cecd-48d6-b19a-3f8479146036/volumes/kubernetes.io~projected/kube-api-access-9zp2t major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/660c1701-2942-48bd-aee9-ff5341e6ffce/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/660c1701-2942-48bd-aee9-ff5341e6ffce/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6f241cd5-d596-4fb1-a270-39d5f3a83727/volumes/kubernetes.io~projected/kube-api-access-zbjn6:{mountpoint:/var/lib/kubelet/pods/6f241cd5-d596-4fb1-a270-39d5f3a83727/volumes/kubernetes.io~projected/kube-api-access-zbjn6 major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae/volumes/kubernetes.io~projected/kube-api-access-z2qvx:{mountpoint:/var/lib/kubelet/pods/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae/volumes/kubernetes.io~projected/kube-api-access-z2qvx major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89ac4cee-c777-4655-9bcc-aa09aa136ab5/volumes/kubernetes.io~projected/kube-api-access-7lt8g:{mountpoint:/var/lib/kubelet/pods/89ac4cee-c777-4655-9bcc-aa09aa136ab5/volumes/kubernetes.io~projected/kube-api-access-7lt8g major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/89ac4cee-c777-4655-9bcc-aa09aa136ab5/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/89ac4cee-c777-4655-9bcc-aa09aa136ab5/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a873a458-b1cc-4b84-9a61-a35b5c783868/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a873a458-b1cc-4b84-9a61-a35b5c783868/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b4e4fdae-7252-4a99-9f43-3939b224757c/volumes/kubernetes.io~projected/kube-api-access-mddmq:{mountpoint:/var/lib/kubelet/pods/b4e4fdae-7252-4a99-9f43-3939b224757c/volumes/kubernetes.io~projected/kube-api-access-mddmq major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2/volumes/kubernetes.io~projected/kube-api-access-nw9mm:{mountpoint:/var/lib/kubelet/pods/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2/volumes/kubernetes.io~projected/kube-api-access-nw9mm major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc469038-9017-4941-972e-0cb0bc3adf6c/volumes/kubernetes.io~projected/kube-api-access-ttttp:{mountpoint:/var/lib/kubelet/pods/bc469038-9017-4941-972e-0cb0bc3adf6c/volumes/kubernetes.io~projected/kube-api-access-ttttp major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c0a001a3-66bc-4816-aa70-7b3c46cbc10f/volumes/kubernetes.io~projected/kube-api-access-phbmw:{mountpoint:/var/lib/kubelet/pods/c0a001a3-66bc-4816-aa70-7b3c46cbc10f/volumes/kubernetes.io~projected/kube-api-access-phbmw major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d05191cb-2a10-4929-8000-b87e80b7070f/volumes/kubernetes.io~projected/kube-api-access-gvwng:{mountpoint:/var/lib/kubelet/pods/d05191cb-2a10-4929-8000-b87e80b7070f/volumes/kubernetes.io~projected/kube-api-access-gvwng major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f14b5c6b-3af4-49b0-a5bb-930d814fa069/volumes/kubernetes.io~projected/kube-api-access-8244x:{mountpoint:/var/lib/kubelet/pods/f14b5c6b-3af4-49b0-a5bb-930d814fa069/volumes/kubernetes.io~projected/kube-api-access-8244x major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volumes/kubernetes.io~projected/kube-api-access-rf5sc:{mountpoint:/var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volumes/kubernetes.io~projected/kube-api-access-rf5sc major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/60da77b3dc611bdd11479abd4384d44df2f51ec76d27ed826350ab622a461ab7/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/a4afa4ca2448871d70425f86c55aa3a488be0f5987f56c8b66841ae915bc332f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/196f8ea9b63e0878d9e0a99e3af310b4f6cc90984edaabdb605867af2dbd120b/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/e08564627d287ce2a036b198a95236784e1411bce32b79d15820a391f4ff949e/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/fefa2aa3ca9a520ace59b0c3c420d468f90765f04562b511404b5b29198cde51/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/37518fdb3f3102e2e4e5dd56197669ea7382f59342a1ccb686c34e2849504d30/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/f1d72ce433346bd968a53da332c8b0b572a2a9c16f911305069f19a1e320bfc1/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/4f3b1afe4ad90425613bdcc54a44d0a95c9dd21325058fa900ffe490059b7f05/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/565491d07dab019f9f9783bb23f5186e2c143f0047c61b19b6d6d82b3dbdee82/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/ba04058cce24c69384af1b70e8240ad52f445a2806dc03e816bac16c75e961a3/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/3763579079c130fa4fb2aa27083d6a598375b8f3f674b3c780676b5d351798e2/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-146:{mountpoint:/var/lib/containers/storage/overlay/049328f70037f92430032e559f3216905c14426bc034247e981dce09a540b72b/merged major:0 minor:146 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/00da7bc54f905bee2d9477daa7b28abd6c12fe9ad0957c3b93e4a733d2a3662f/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/9967dd833db64883b4cdf1059e33b57e5766dc5496ddb3d75c8f9d30ce14622f/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/550c6b75ae035399445123d2360d89167522aa4b1e64ddb98b269ec342458845/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/f04bbbae8088f28ad9de7d42e3d6baeb87221a1974bc633c23f2e8d456e0f2db/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/28cdc1b88e1fe5a4878f6a333f20b8b66cdf1591c9d4ab1550694705fdd6bdb6/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/293e65214b0d6b29bc4dc4cc37b9497ba7ce55080cb2a7ed4e96ad4fe63db8d4/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/bd89ad25c74e99012f9c577b04582b69d235d59c9ddcd865732dc1f78b20c296/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/b6c43b27546f84d3c5eec376c27d8da638e2e711f30ebed7369a086143deaff9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/95755dbe10e34b777fc0a32476312a11ceac0b4c1f532adadf97dd78a24b7d40/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/529a4acb01e652a9204fadeefc67d8f124cf772bf5a2f0c567a01efb67a08c07/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/9b1a3ae6cdcd9c99e946198693b90a26e82ac7657b269c71e192130940644900/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/4a64ac9416fc00c06065043b944b2e4191ece4596701bffe1de0bfd77d5d4903/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/3ff22e121f7dce69f8b6a59f86e572cfb7949016a99c4f0c4bae953266e3703f/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/e684096394832482b41aeb8b1444ee3bdfa88a842b09d85ff382c441d54bded6/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/aac03644d1d01b11238753ecf17d45e7105da48dd0c899617dc258c17506d70f/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/b38f81e0fdf875694c4d52749653cfb7a2ba31120e4aa640baaeffe35e9fa727/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/adc0a2943f0b401d90b73eaa64a895b31f7d2f7920e93e0c4144f74e98848884/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/ac1d6bd89ca55461a570ad62cec9fc81c181ae95366a2e31af7a313edff647bc/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-50:{mountpoint:/var/lib/containers/storage/overlay/ff7d3cf666dc5dd2b37c6c0652d0ca2ed07582a5f1a4d20cb421a4d3309dbe17/merged major:0 minor:50 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/8a5f4423a583b634eeb584aa230ff6a2bebef5c838f97c13c5e992cd0bfafb43/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/b76dbabd63c6438124c000610ba1cdee14d36e12e4366a55313f8aea49a35b04/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/b39618701a6f726915c3d875499e41f1287798672ca9bd4f2c203ef619dbd806/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/81510da306434aae274ba72b773dc3411dbeda7a5428b209a92aeb2aeb392d1c/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c76732f5b27ea7fb0b463df96c914761348e260380cadafa4f3d5b338c314a0d/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/cecd4d9e1a08117900b489501e1f500622d8727befdafc93b66bbd033ad5b463/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/22212d87937f13088c920ec78b2c5181a6d1cc5f559c7cdf535da245cf81ac8f/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/8335c170eb6d95e6f7a3134c05a22e4fffb37c544c6bdd4485ba61774320c757/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/af8581567301b3b4b6af8e763cd401dd7a81dc0e45f768382309d77316242236/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.052812    4478 manager.go:217] Machine: {Timestamp:2026-10-31 00:02:47.052246298 +0000 UTC m=+0.068911000 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100492 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a751e8bdb771810dd8dfca4615406 SystemUUID:ec2a751e-8bdb-7718-10dd-8dfca4615406 BootID:e646b4b4-9bf8-448a-bc93-caff5599036c Filesystems:[{Device:/var/lib/kubelet/pods/3ffb287c-6e5e-4adb-9278-b26eec222aeb/volumes/kubernetes.io~projected/kube-api-access-fd8tq DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2/volumes/kubernetes.io~projected/kube-api-access-nw9mm DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/14418f52-35b7-4c4e-9593-cc99bedcb693/volumes/kubernetes.io~projected/kube-api-access-fhzm8 DeviceMajor:0 DeviceMinor:240 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa131176a24d7a480616921a8f7d18a2849c3600431ed41f33564afdcbf3bb62/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9ac9ec404834297b9aacf260e4c746147557dbd2d84004a3de46b8b43bfa337e/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/74a7a038a75ede711a77e1a1db154e45374f15091e12442eb31160ef6381ff5b/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-146 DeviceMajor:0 DeviceMinor:146 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/85f3547bef7bbc8accbdbbf054969329f4d52331eaee88cbe803fb8ba770b60b/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d05191cb-2a10-4929-8000-b87e80b7070f/volumes/kubernetes.io~projected/kube-api-access-gvwng DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ec8cff04bfc073f3a7e6fa3d336449b06b9f9b05831b1df71f9c3375f43acc7f/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c0a001a3-66bc-4816-aa70-7b3c46cbc10f/volumes/kubernetes.io~projected/kube-api-access-phbmw DeviceMajor:0 DeviceMinor:222 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89ac4cee-c777-4655-9bcc-aa09aa136ab5/volumes/kubernetes.io~projected/kube-api-access-7lt8g DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/660c1701-2942-48bd-aee9-ff5341e6ffce/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/bc469038-9017-4941-972e-0cb0bc3adf6c/volumes/kubernetes.io~projected/kube-api-access-ttttp DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae/volumes/kubernetes.io~projected/kube-api-access-z2qvx DeviceMajor:0 DeviceMinor:230 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-50 DeviceMajor:0 DeviceMinor:50 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19be463a-c645-48ab-a633-42fb816ad71c/volumes/kubernetes.io~projected/kube-api-access-gp4rs DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/544adb5d4fefa8c085ffa0508ef96666fd312b2293e404898d9459206b9f3638/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/49220507-cecd-48d6-b19a-3f8479146036/volumes/kubernetes.io~projected/kube-api-access-9zp2t DeviceMajor:0 DeviceMinor:213 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/371b6af986371e6ca375618ca8c5247f1f70184e27f95c2d825347c77fc74ce1/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6f241cd5-d596-4fb1-a270-39d5f3a83727/volumes/kubernetes.io~projected/kube-api-access-zbjn6 DeviceMajor:0 DeviceMinor:223 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/14418f52-35b7-4c4e-9593-cc99bedcb693/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:239 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e165600f0b6c1b8bc68712d4ccdb0d75ac43ed3c137856a896199d5484209cc6/userdata/shm DeviceMajor:0 DeviceMinor:79 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9e853f16ab0f1d341e1418e38cf7c509b8621c9aa6a33389f02c1241005ae304/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1a6de299-a94e-4591-acac-b5018a133c65/volumes/kubernetes.io~projected/kube-api-access-lvb88 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/89ac4cee-c777-4655-9bcc-aa09aa136ab5/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/433c6858-00d2-47fc-8e98-9df265e3ce28/volumes/kubernetes.io~projected/kube-api-access-s5z6b DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d52f1ed7d7ccc1c6a80b82236bb6fbc125c65f65464fddec0f7168505076abc3/userdata/shm DeviceMajor:0 DeviceMinor:91 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b4e4fdae-7252-4a99-9f43-3939b224757c/volumes/kubernetes.io~projected/kube-api-access-mddmq DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a873a458-b1cc-4b84-9a61-a35b5c783868/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e6983441c9681e6c333acacd7f5f9169b876b96abb004314c7975bd01ba1cd63/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fbeabba14eee4c434f016955203e6cbf3e2cb0093de51c0fcf1ba17eefd7a1e8/userdata/shm DeviceMajor:0 DeviceMinor:235 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f14b5c6b-3af4-49b0-a5bb-930d814fa069/volumes/kubernetes.io~projected/kube-api-access-8244x DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/562b2fbeeb37abca73e132cdc9145a628789c5d8056f4ac3be5d56f650f5adef/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/f9cde72b-3aa1-4c45-95e1-72a55236dc1e/volumes/kubernetes.io~projected/kube-api-access-rf5sc DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:07385e3cdb25aa1 MacAddress:be:d2:29:e2:0e:11 Speed:10000 Mtu:8901} {Name:74a7a038a75ede7 MacAddress:fe:ca:b2:fa:9f:dc Speed:10000 Mtu:8901} {Name:85f3547bef7bbc8 MacAddress:8a:91:09:8f:99:d7 Speed:10000 Mtu:8901} {Name:9e853f16ab0f1d3 MacAddress:da:ac:ae:8f:ab:98 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:48:ac:79:97:b7 Speed:0 Mtu:9001} {Name:br-int MacAddress:9e:63:d9:a4:8a:3e Speed:0 Mtu:8901} {Name:e6983441c9681e6 MacAddress:aa:f8:53:d0:ef:af Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:48:ac:79:97:b7 Speed:0 Mtu:9001} {Name:fbeabba14eee4c4 MacAddress:02:09:11:c0:8d:0f Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:b6:61:b4:ba:c1:97 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:7a:62:28:89:bf:96 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053101    4478 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053146    4478 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053438    4478 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053453    4478 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-148.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053575    4478 topology_manager.go:143] "Creating topology manager with none policy"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053583    4478 container_manager_linux.go:311] "Creating device plugin manager"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053597    4478 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053614    4478 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053684    4478 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053839    4478 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.053966    4478 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.054006    4478 kubelet.go:493] "Attempting to sync node with API server"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.054019    4478 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.054029    4478 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.054039    4478 kubelet.go:405] "Adding apiserver pod source"
Oct 31 00:02:47.054550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.054047    4478 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 31 00:02:47.057029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.057011    4478 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:02:47.057029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.057028    4478 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:02:47.059554 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.057518    4478 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 31 00:02:47.064649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.064632    4478 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 31 00:02:47.064649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.064647    4478 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 31 00:02:47.064765 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.064749    4478 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 31 00:02:47.064964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.064952    4478 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 31 00:02:47.065004 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.064982    4478 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 31 00:02:47.065143 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065117    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065148    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065157    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065162    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065168    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065174    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065179    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065185    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 31 00:02:47.065190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065191    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 31 00:02:47.065469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065197    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 31 00:02:47.065469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065212    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 31 00:02:47.065469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065221    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 31 00:02:47.065469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065245    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 31 00:02:47.065469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.065251    4478 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 31 00:02:47.068849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.068834    4478 server.go:1263] "Started kubelet"
Oct 31 00:02:47.069282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.069219    4478 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 31 00:02:47.069366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.069307    4478 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 31 00:02:47.069780 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.069761    4478 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.070536    4478 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071500    4478 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071751    4478 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071789    4478 server.go:317] "Adding debug handlers to kubelet server"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071826    4478 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071827    4478 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:57:09 +0000 UTC" deadline="2028-07-01 08:30:54.601006462 +0000 UTC"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071855    4478 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071860    4478 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14624h28m7.529149768s"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.071951    4478 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.071970    4478 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.072819    4478 factory.go:55] Registering systemd factory
Oct 31 00:02:47.073009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.072839    4478 factory.go:223] Registration of the systemd container factory successfully
Oct 31 00:02:47.073672 ip-10-0-137-148 systemd[1]: Started Kubernetes Kubelet.
Oct 31 00:02:47.074163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.074069    4478 factory.go:153] Registering CRI-O factory
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.075563    4478 factory.go:223] Registration of the crio container factory successfully
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.075619    4478 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.075644    4478 factory.go:103] Registering Raw factory
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.075660    4478 manager.go:1199] Started watching for new ooms in manager
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.075987    4478 manager.go:319] Starting recovery of all containers
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.075953    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a873a458-b1cc-4b84-9a61-a35b5c783868" volumeName="kubernetes.io/configmap/a873a458-b1cc-4b84-9a61-a35b5c783868-konnectivity-ca" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076044    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9cde72b-3aa1-4c45-95e1-72a55236dc1e" volumeName="kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-script-lib" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076059    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1a6de299-a94e-4591-acac-b5018a133c65" volumeName="kubernetes.io/projected/1a6de299-a94e-4591-acac-b5018a133c65-kube-api-access-lvb88" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076079    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ffb287c-6e5e-4adb-9278-b26eec222aeb" volumeName="kubernetes.io/projected/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kube-api-access-fd8tq" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076092    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c6858-00d2-47fc-8e98-9df265e3ce28" volumeName="kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-tuned" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076109    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f241cd5-d596-4fb1-a270-39d5f3a83727" volumeName="kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076124    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a001a3-66bc-4816-aa70-7b3c46cbc10f" volumeName="kubernetes.io/projected/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-api-access-phbmw" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076136    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14b5c6b-3af4-49b0-a5bb-930d814fa069" volumeName="kubernetes.io/projected/f14b5c6b-3af4-49b0-a5bb-930d814fa069-kube-api-access-8244x" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076155    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05191cb-2a10-4929-8000-b87e80b7070f" volumeName="kubernetes.io/configmap/d05191cb-2a10-4929-8000-b87e80b7070f-serviceca" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076167    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9cde72b-3aa1-4c45-95e1-72a55236dc1e" volumeName="kubernetes.io/projected/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-kube-api-access-rf5sc" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076185    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" volumeName="kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076198    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" volumeName="kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076215    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9cde72b-3aa1-4c45-95e1-72a55236dc1e" volumeName="kubernetes.io/secret/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076228    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" volumeName="kubernetes.io/projected/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-kube-api-access-nw9mm" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076251    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14418f52-35b7-4c4e-9593-cc99bedcb693" volumeName="kubernetes.io/secret/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-key" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076263    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c6858-00d2-47fc-8e98-9df265e3ce28" volumeName="kubernetes.io/projected/433c6858-00d2-47fc-8e98-9df265e3ce28-kube-api-access-s5z6b" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076276    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89ac4cee-c777-4655-9bcc-aa09aa136ab5" volumeName="kubernetes.io/projected/89ac4cee-c777-4655-9bcc-aa09aa136ab5-kube-api-access-7lt8g" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076294    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a873a458-b1cc-4b84-9a61-a35b5c783868" volumeName="kubernetes.io/secret/a873a458-b1cc-4b84-9a61-a35b5c783868-agent-certs" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076306    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e4fdae-7252-4a99-9f43-3939b224757c" volumeName="kubernetes.io/empty-dir/b4e4fdae-7252-4a99-9f43-3939b224757c-tmp-dir" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076324    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19be463a-c645-48ab-a633-42fb816ad71c" volumeName="kubernetes.io/configmap/19be463a-c645-48ab-a633-42fb816ad71c-iptables-alerter-script" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076336    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f241cd5-d596-4fb1-a270-39d5f3a83727" volumeName="kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076350    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f241cd5-d596-4fb1-a270-39d5f3a83727" volumeName="kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076367    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6f241cd5-d596-4fb1-a270-39d5f3a83727" volumeName="kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076398    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e4fdae-7252-4a99-9f43-3939b224757c" volumeName="kubernetes.io/configmap/b4e4fdae-7252-4a99-9f43-3939b224757c-config-volume" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076786    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b4e4fdae-7252-4a99-9f43-3939b224757c" volumeName="kubernetes.io/projected/b4e4fdae-7252-4a99-9f43-3939b224757c-kube-api-access-mddmq" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076817    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" volumeName="kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076831    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc469038-9017-4941-972e-0cb0bc3adf6c" volumeName="kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-cni-binary-copy" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076872    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14418f52-35b7-4c4e-9593-cc99bedcb693" volumeName="kubernetes.io/projected/14418f52-35b7-4c4e-9593-cc99bedcb693-kube-api-access-fhzm8" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076903    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19be463a-c645-48ab-a633-42fb816ad71c" volumeName="kubernetes.io/projected/19be463a-c645-48ab-a633-42fb816ad71c-kube-api-access-gp4rs" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076918    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc469038-9017-4941-972e-0cb0bc3adf6c" volumeName="kubernetes.io/projected/bc469038-9017-4941-972e-0cb0bc3adf6c-kube-api-access-ttttp" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076934    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c0a001a3-66bc-4816-aa70-7b3c46cbc10f" volumeName="kubernetes.io/empty-dir/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-data-volume" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076947    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d05191cb-2a10-4929-8000-b87e80b7070f" volumeName="kubernetes.io/projected/d05191cb-2a10-4929-8000-b87e80b7070f-kube-api-access-gvwng" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076959    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f14b5c6b-3af4-49b0-a5bb-930d814fa069" volumeName="kubernetes.io/empty-dir/f14b5c6b-3af4-49b0-a5bb-930d814fa069-tmp-dir" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076982    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9cde72b-3aa1-4c45-95e1-72a55236dc1e" volumeName="kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-env-overrides" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.076995    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f9cde72b-3aa1-4c45-95e1-72a55236dc1e" volumeName="kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-config" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077013    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="433c6858-00d2-47fc-8e98-9df265e3ce28" volumeName="kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-tmp" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077079    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="49220507-cecd-48d6-b19a-3f8479146036" volumeName="kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077094    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="660c1701-2942-48bd-aee9-ff5341e6ffce" volumeName="kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077115    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="89ac4cee-c777-4655-9bcc-aa09aa136ab5" volumeName="kubernetes.io/secret/89ac4cee-c777-4655-9bcc-aa09aa136ab5-cert" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077149    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc469038-9017-4941-972e-0cb0bc3adf6c" volumeName="kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-daemon-config" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077171    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14418f52-35b7-4c4e-9593-cc99bedcb693" volumeName="kubernetes.io/configmap/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-cabundle" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077186    4478 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="868e0cf5-15d2-4a00-9fe6-01ff90ac0aae" volumeName="kubernetes.io/projected/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae-kube-api-access-z2qvx" seLinuxMountContext=""
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077199    4478 reconstruct.go:97] "Volume reconstruction finished"
Oct 31 00:02:47.083423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.077207    4478 reconciler.go:29] "Reconciler: start to sync state"
Oct 31 00:02:47.107964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.107938    4478 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 31 00:02:47.109856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.109498    4478 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 31 00:02:47.109856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.109522    4478 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 31 00:02:47.109856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.109544    4478 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 31 00:02:47.109856 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.109588    4478 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 31 00:02:47.151745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.151725    4478 manager.go:324] Recovery completed
Oct 31 00:02:47.152628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.152582    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="f39f539f9d9d8952b5abd7d2d0bc308366bd2a267f1082dbc5cf4125b1e84167" exitCode=0
Oct 31 00:02:47.152628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.152618    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="7043b161f2be064558706c8c3eba856b6dfdfcf2f75f6c032672e7c2e629dcf2" exitCode=0
Oct 31 00:02:47.152628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.152625    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="8e7fbb10c7a14dbe06340268df6ee1a317b9dfe5ebd75b2a765f0d16cd487ce1" exitCode=0
Oct 31 00:02:47.152628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.152632    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="e4ee70b6cb814d087991ee8729d166b11b991a4dd66787f28f695b4569f1810b" exitCode=0
Oct 31 00:02:47.152886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.152638    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="4e532db717b632feb6164721fe0bd3a3673344ce5026412da077aaeb08264c44" exitCode=0
Oct 31 00:02:47.152886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.152643    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2" containerID="017a9e0577d43412556df290eb7675d974204fb2b4944acff685f861307075ac" exitCode=0
Oct 31 00:02:47.154662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.154632    4478 generic.go:356] "Generic (PLEG): container finished" podID="f7aae722a65c2075f78dd611b7e7f739" containerID="ee0c8006630b804274d38b45146f13f80f48cb3bea2e94ca3c4fca404422c866" exitCode=0
Oct 31 00:02:47.169667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.169612    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.170758 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.170739    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.170851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.170767    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.170851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.170783    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.172044 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.172022    4478 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:47.172343 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172323    4478 cpu_manager.go:226] "Starting" policy="none"
Oct 31 00:02:47.172343 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172338    4478 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 31 00:02:47.172511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172358    4478 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 31 00:02:47.172511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172364    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172511    4478 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172525    4478 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172547    4478 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172556    4478 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172564    4478 policy_none.go:50] "Start"
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172574    4478 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 31 00:02:47.172604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172586    4478 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 31 00:02:47.172861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172738    4478 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 31 00:02:47.172861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172748    4478 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 31 00:02:47.172861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.172755    4478 policy_none.go:44] "Start"
Oct 31 00:02:47.173267 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.173247    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.173348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.173278    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.173348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.173293    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.178433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.178404    4478 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 31 00:02:47.183680 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.183662    4478 manager.go:341] "Starting Device Plugin manager"
Oct 31 00:02:47.183772 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.183688    4478 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 31 00:02:47.183772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.183698    4478 server.go:87] "Starting device plugin registration server"
Oct 31 00:02:47.183999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.183986    4478 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 31 00:02:47.184082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.184000    4478 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 31 00:02:47.186521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.185780    4478 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 31 00:02:47.186521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.185869    4478 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 31 00:02:47.186521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.185906    4478 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 31 00:02:47.186521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.185913    4478 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 31 00:02:47.186521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.185974    4478 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-31T00:02:47.185896673Z","UUID":"454c2527-50ad-4264-9938-4d81e6fb9d95","Handler":null,"Name":"","Endpoint":""}
Oct 31 00:02:47.187534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.187514    4478 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 31 00:02:47.187638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.187551    4478 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 31 00:02:47.187638 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.187512    4478 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 31 00:02:47.187638 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.187600    4478 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-148.us-west-1.compute.internal\" not found"
Oct 31 00:02:47.210542 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.210494    4478 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"]
Oct 31 00:02:47.210655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.210640    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.211635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.211616    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.211708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.211652    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.211708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.211667    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.211972 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.211953    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.212134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.212120    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.212965 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.212918    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.212965 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.212945    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.213102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.212970    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.213102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.212985    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.213102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.212950    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.213102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213053    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.213245 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.213143    4478 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.213372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213335    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"3031cca7be5520d5a22eb64a065c3c72","Type":"ContainerStarted","Data":"7c699fcc7440f68d9a26851c3d71e085a03a8a388085fc79a8399e7d4f89d6d5"}
Oct 31 00:02:47.213461 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213402    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"3031cca7be5520d5a22eb64a065c3c72","Type":"ContainerStarted","Data":"e20f0d5d825feb22f97288f21eb9858fcf387dfc9e4b512db74120ce1b24df05"}
Oct 31 00:02:47.213497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213461    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"f7aae722a65c2075f78dd611b7e7f739","Type":"ContainerStarted","Data":"c248e2a1d2ce0bc4de9bfb788ed5cc55d89cc45563d726e7a7ad5fbee159ba22"}
Oct 31 00:02:47.213497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213488    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"f7aae722a65c2075f78dd611b7e7f739","Type":"ContainerDied","Data":"ee0c8006630b804274d38b45146f13f80f48cb3bea2e94ca3c4fca404422c866"}
Oct 31 00:02:47.213557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213506    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal" event={"ID":"f7aae722a65c2075f78dd611b7e7f739","Type":"ContainerStarted","Data":"bc502ec19ed351ff55aebfd2298d752ebfb0c09972142436c7111b4db5260dc8"}
Oct 31 00:02:47.213557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.213467    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.214360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.214345    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.214467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.214374    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.214467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.214411    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.214585 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.214567    4478 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-148.us-west-1.compute.internal\" not found" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.222769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.222751    4478 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:47.267837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.267805    4478 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:47.269870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.269837    4478 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:47.272091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.272078    4478 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.284446 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.284432    4478 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 31 00:02:47.285314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.285299    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientMemory"
Oct 31 00:02:47.285400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.285328    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasNoDiskPressure"
Oct 31 00:02:47.285400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.285338    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeHasSufficientPID"
Oct 31 00:02:47.285400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.285358    4478 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.319630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.319612    4478 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 31 00:02:47.322206 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.322193    4478 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 31 00:02:47.333940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.333924    4478 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:02:47.334020 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.333971    4478 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.334020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.333983    4478 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.418718 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.418691    4478 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.418785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.418747    4478 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.419207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.419192    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeNotReady"
Oct 31 00:02:47.419268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.419221    4478 setters.go:546] "Node became not ready" node="ip-10-0-137-148.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-31T00:02:47Z","lastTransitionTime":"2026-10-31T00:02:47Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 31 00:02:47.422400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.422331    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.422400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.422375    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.422485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.422454    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3031cca7be5520d5a22eb64a065c3c72-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"3031cca7be5520d5a22eb64a065c3c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.440552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.440533    4478 nodeinfomanager.go:471] existing CSINode "ip-10-0-137-148.us-west-1.compute.internal" is owned by different node (oldNodeID="94911e8f-831d-425c-a4b4-d399a2804eb5", newNodeID=""), cleaning up...
Oct 31 00:02:47.443917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.443904    4478 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:02:47.443953 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:47.443936    4478 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.522892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.522872    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.522974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.522896    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.522974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.522915    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3031cca7be5520d5a22eb64a065c3c72-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"3031cca7be5520d5a22eb64a065c3c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.522974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.522965    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.523058 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.522982    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f7aae722a65c2075f78dd611b7e7f739-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"f7aae722a65c2075f78dd611b7e7f739\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:47.523058 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:47.522965    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3031cca7be5520d5a22eb64a065c3c72-config\") pod \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" (UID: \"3031cca7be5520d5a22eb64a065c3c72\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:48.055512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.055488    4478 apiserver.go:52] "Watching apiserver"
Oct 31 00:02:48.162307 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.162275    4478 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 31 00:02:48.162873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.162847    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal","openshift-multus/multus-pqz2l","openshift-image-registry/node-ca-f9k85","openshift-ovn-kubernetes/ovnkube-node-s95vh","openshift-dns/dns-default-hlhj6","openshift-insights/periodic-gathering-4xtcf-rd6pb","openshift-network-diagnostics/network-check-target-s64vg","openshift-dns/node-resolver-chjbs","kube-system/global-pull-secret-syncer-gfzr6","openshift-multus/network-metrics-daemon-9cbv5","openshift-cluster-node-tuning-operator/tuned-mnzps","openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4","openshift-network-operator/iptables-alerter-9bwpb","kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal","openshift-ingress-canary/ingress-canary-gjbb9","openshift-service-ca/service-ca-5fffc97bdf-bkzrc","openshift-insights/insights-runtime-extractor-mvljb","kube-system/konnectivity-agent-cprhd","openshift-multus/multus-additional-cni-plugins-9b5br","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"]
Oct 31 00:02:48.162962 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.162947    4478 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:48.163140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.163127    4478 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:48.166194 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.166175    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:48.169554 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.169479    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.170880 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.170864    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.172991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.172976    4478 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 31 00:02:48.216351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.216332    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zgjmv\""
Oct 31 00:02:48.217081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.217064    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 31 00:02:48.217167 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.217111    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.217358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.217340    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 31 00:02:48.217479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.217365    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.217479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.217392    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.217479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.217419    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.219009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.218996    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 31 00:02:48.224870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.224858    4478 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:02:48.224912 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.224894    4478 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:48.225326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.225315    4478 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 31 00:02:48.225356 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.225342    4478 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-148.us-west-1.compute.internal"
Oct 31 00:02:48.227560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227539    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-k8s-cni-cncf-io\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.227603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227574    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phbmw\" (UniqueName: \"kubernetes.io/projected/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-api-access-phbmw\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.227603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227590    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-systemd-units\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.227690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227606    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-bin\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.227690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227622    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-netd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.227690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227676    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysconfig\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.227802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227726    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-systemd\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.227802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227763    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5z6b\" (UniqueName: \"kubernetes.io/projected/433c6858-00d2-47fc-8e98-9df265e3ce28-kube-api-access-s5z6b\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.227802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227794    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7lt8g\" (UniqueName: \"kubernetes.io/projected/89ac4cee-c777-4655-9bcc-aa09aa136ab5-kube-api-access-7lt8g\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:48.227911 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227831    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.227991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227973    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nw9mm\" (UniqueName: \"kubernetes.io/projected/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-kube-api-access-nw9mm\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.228044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.227999    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-run\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.228114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228097    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-system-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.228161 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228122    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-multus-certs\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.228211 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228159    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttttp\" (UniqueName: \"kubernetes.io/projected/bc469038-9017-4941-972e-0cb0bc3adf6c-kube-api-access-ttttp\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.228211 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228179    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19be463a-c645-48ab-a633-42fb816ad71c-iptables-alerter-script\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.228211 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228195    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-slash\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.228348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228212    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e4fdae-7252-4a99-9f43-3939b224757c-tmp-dir\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.228348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228220    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.228348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228228    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-cnibin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.228348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228304    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b4e4fdae-7252-4a99-9f43-3939b224757c-tmp-dir\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.228553 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228403    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.228553 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228432    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a873a458-b1cc-4b84-9a61-a35b5c783868-agent-certs\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:48.228553 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228496    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d05191cb-2a10-4929-8000-b87e80b7070f-serviceca\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.228553 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228529    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:48.228685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228557    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.228685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228576    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-kubelet\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.228685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228591    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-daemon-config\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.228685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228613    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.228685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228644    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-crio-socket\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228756    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228787    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-script-lib\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228812    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228839    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qvx\" (UniqueName: \"kubernetes.io/projected/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae-kube-api-access-z2qvx\") pod \"migrator-66df5d496b-db4f4\" (UID: \"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228862    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-kubernetes\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228887    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-sys\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.228938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228919    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-cni-binary-copy\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.228950    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-system-cni-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229004    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-config\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229042    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovn-node-metrics-cert\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229094    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229114    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-cabundle\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229153    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-modprobe-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.229208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229174    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229210    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-config\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229224    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229247    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-netns\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229209    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-cni-binary-copy\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229264    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4fdae-7252-4a99-9f43-3939b224757c-config-volume\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229288    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8tq\" (UniqueName: \"kubernetes.io/projected/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kube-api-access-fd8tq\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229313    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19be463a-c645-48ab-a633-42fb816ad71c-host-slash\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229339    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cnibin\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229356    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-kubelet\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229369    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-hostroot\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229413    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-conf-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229436    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gp4rs\" (UniqueName: \"kubernetes.io/projected/19be463a-c645-48ab-a633-42fb816ad71c-kube-api-access-gp4rs\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.229513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229494    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8244x\" (UniqueName: \"kubernetes.io/projected/f14b5c6b-3af4-49b0-a5bb-930d814fa069-kube-api-access-8244x\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229564    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89ac4cee-c777-4655-9bcc-aa09aa136ab5-cert\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229618    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229684    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-var-lib-kubelet\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229720    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-etc-kubernetes\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229737    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-var-lib-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229761    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-etc-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229787    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-env-overrides\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229819    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229882    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mddmq\" (UniqueName: \"kubernetes.io/projected/b4e4fdae-7252-4a99-9f43-3939b224757c-kube-api-access-mddmq\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229911    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229962    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-tuned\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.229994    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230031    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-tuned\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230063    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230083    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-host\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230116    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-bin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230136    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-node-log\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230153    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvb88\" (UniqueName: \"kubernetes.io/projected/1a6de299-a94e-4591-acac-b5018a133c65-kube-api-access-lvb88\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230169    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230200    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-key\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230254    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cni-binary-copy\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230276    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230315    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-tmp\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230343    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-socket-dir-parent\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230357    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f14b5c6b-3af4-49b0-a5bb-930d814fa069-hosts-file\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230371    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05191cb-2a10-4929-8000-b87e80b7070f-host\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230402    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-ovn\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230481    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/433c6858-00d2-47fc-8e98-9df265e3ce28-tmp\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230492    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-log-socket\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.230556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230534    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-dbus\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230573    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230596    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-multus\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230626    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230662    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-device-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230686    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-conf\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230741    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-systemd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230783    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230807    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230837    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-kubelet-config\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230860    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhzm8\" (UniqueName: \"kubernetes.io/projected/14418f52-35b7-4c4e-9593-cc99bedcb693-kube-api-access-fhzm8\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230876    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-netns\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230892    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230908    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-data-volume\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230923    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14b5c6b-3af4-49b0-a5bb-930d814fa069-tmp-dir\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230953    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a873a458-b1cc-4b84-9a61-a35b5c783868-konnectivity-ca\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230982    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvwng\" (UniqueName: \"kubernetes.io/projected/d05191cb-2a10-4929-8000-b87e80b7070f-kube-api-access-gvwng\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.230995    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-data-volume\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.231003    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f14b5c6b-3af4-49b0-a5bb-930d814fa069-tmp-dir\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.231013    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.231004    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.231050    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf5sc\" (UniqueName: \"kubernetes.io/projected/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-kube-api-access-rf5sc\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.231178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.231067    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-lib-modules\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.287449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.287429    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nw9mm\" (UniqueName: \"kubernetes.io/projected/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-kube-api-access-nw9mm\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.290015 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.289992    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttttp\" (UniqueName: \"kubernetes.io/projected/bc469038-9017-4941-972e-0cb0bc3adf6c-kube-api-access-ttttp\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.290183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.290168    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5z6b\" (UniqueName: \"kubernetes.io/projected/433c6858-00d2-47fc-8e98-9df265e3ce28-kube-api-access-s5z6b\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.292581 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.292566    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvb88\" (UniqueName: \"kubernetes.io/projected/1a6de299-a94e-4591-acac-b5018a133c65-kube-api-access-lvb88\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:48.316731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.316674    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 31 00:02:48.316810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.316793    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 31 00:02:48.316855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.316824    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 31 00:02:48.317141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317125    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 31 00:02:48.317191 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317152    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 31 00:02:48.317477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317420    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 31 00:02:48.317594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317570    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-j7lc6\""
Oct 31 00:02:48.317594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317588    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hplth\""
Oct 31 00:02:48.317683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317598    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5ddpr\""
Oct 31 00:02:48.317683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317590    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-tqqqs\""
Oct 31 00:02:48.317683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317590    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vtt85\""
Oct 31 00:02:48.317683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.317615    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.319223 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319204    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.319258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319228    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 31 00:02:48.319258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319240    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovn-node-metrics-cert\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.319362 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319349    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 31 00:02:48.319456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319440    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.319456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319448    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-hkxnt\""
Oct 31 00:02:48.319575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319457    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-m4gwp\""
Oct 31 00:02:48.319575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319449    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 31 00:02:48.319656 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319599    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/b4e4fdae-7252-4a99-9f43-3939b224757c-config-volume\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.319719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319703    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zm4vk\""
Oct 31 00:02:48.319796 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319748    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.319860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319845    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/89ac4cee-c777-4655-9bcc-aa09aa136ab5-cert\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:48.319905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319864    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 31 00:02:48.319905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.319875    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-hg87m\""
Oct 31 00:02:48.320984 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.320967    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-key\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:48.321396 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.321370    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-fsk29\""
Oct 31 00:02:48.321540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.321501    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.321540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.321505    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.328518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.328500    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7lt8g\" (UniqueName: \"kubernetes.io/projected/89ac4cee-c777-4655-9bcc-aa09aa136ab5-kube-api-access-7lt8g\") pod \"ingress-canary-gjbb9\" (UID: \"89ac4cee-c777-4655-9bcc-aa09aa136ab5\") " pod="openshift-ingress-canary/ingress-canary-gjbb9"
Oct 31 00:02:48.329329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.329313    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/14418f52-35b7-4c4e-9593-cc99bedcb693-signing-cabundle\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:48.331507 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331490    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-netd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.331578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331521    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysconfig\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.331578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331546    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-systemd\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.331676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331592    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-netd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.331676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331605    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysconfig\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.331676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331610    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-systemd\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.331676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331639    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-run\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.331676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331663    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-system-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331688    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-multus-certs\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331712    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-run\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-slash\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331740    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-multus-certs\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331755    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-cnibin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331760    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-slash\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331761    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-system-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331780    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331797    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331813    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-kubelet\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331828    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-cnibin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331836    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-crio-socket\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.331843    4478 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:02:48.331869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331862    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-kubelet\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.332276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331873    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-registration-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.332276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.331959    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-crio-socket\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.332361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332292    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.332361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332330    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.332361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332320    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.332472 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.332361    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:48.832322766 +0000 UTC m=+1.848987461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : configmap "kube-rbac-proxy" not found
Oct 31 00:02:48.332472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332417    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-kubernetes\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.332472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332419    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-sys-fs\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.332472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332440    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-sys\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.332472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332460    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-system-cni-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332476    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332487    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-kubernetes\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332491    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-modprobe-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332508    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-sys\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332517    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-system-cni-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332537    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-netns\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332557    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-modprobe-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.332569    4478 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332604    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-run-netns\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332611    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19be463a-c645-48ab-a633-42fb816ad71c-host-slash\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.332645    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:48.832627558 +0000 UTC m=+1.849292262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:48.332648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332570    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/19be463a-c645-48ab-a633-42fb816ad71c-host-slash\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332668    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cnibin\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332682    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-kubelet\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332695    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-hostroot\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332710    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-conf-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332725    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332739    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-var-lib-kubelet\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332754    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-etc-kubernetes\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332761    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-kubelet\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332774    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-conf-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332778    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-var-lib-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332802    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-cnibin\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332823    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-etc-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332827    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-etc-kubernetes\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332792    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-hostroot\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332844    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-var-lib-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332820    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-var-lib-kubelet\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332868    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-d\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332876    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332873    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-etc-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.332938    4478 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.332969    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:48.832958777 +0000 UTC m=+1.849623467 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : secret "metrics-daemon-secret" not found
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.332992    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333013    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333028    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-host\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333060    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-bin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333090    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-socket-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333100    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-node-log\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333105    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-bin\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333105    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-cni-dir\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333112    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-host\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333170    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333205    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-node-log\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333229    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-socket-dir-parent\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333277    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kubelet-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333301    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f14b5c6b-3af4-49b0-a5bb-930d814fa069-hosts-file\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333313    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-socket-dir-parent\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333331    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05191cb-2a10-4929-8000-b87e80b7070f-host\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333356    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-ovn\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333402    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-log-socket\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333408    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d05191cb-2a10-4929-8000-b87e80b7070f-host\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333407    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f14b5c6b-3af4-49b0-a5bb-930d814fa069-hosts-file\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333430    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-dbus\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333457    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-log-socket\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333432    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-ovn\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333490    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333517    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-multus\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333493    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-dbus\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333544    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333571    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-var-lib-cni-multus\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333575    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-device-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333594    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-conf\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333605    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-etc-selinux\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333610    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-systemd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333640    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3ffb287c-6e5e-4adb-9278-b26eec222aeb-device-dir\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333642    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333664    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-openvswitch\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333664    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.333685    4478 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333713    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-kubelet-config\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333723    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-run-systemd\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333730    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-netns\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333766    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-netns\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333773    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/660c1701-2942-48bd-aee9-ff5341e6ffce-kubelet-config\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333793    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333683    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333838    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-etc-sysctl-conf\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.333853    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:48.833834832 +0000 UTC m=+1.850499538 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333851    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333886    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-lib-modules\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333911    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-k8s-cni-cncf-io\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333942    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/bc469038-9017-4941-972e-0cb0bc3adf6c-host-run-k8s-cni-cncf-io\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333944    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-systemd-units\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333970    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-systemd-units\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333973    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/433c6858-00d2-47fc-8e98-9df265e3ce28-lib-modules\") pod \"tuned-mnzps\" (UID: \"433c6858-00d2-47fc-8e98-9df265e3ce28\") " pod="openshift-cluster-node-tuning-operator/tuned-mnzps"
Oct 31 00:02:48.333953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.333987    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-bin\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.335335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.334041    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-host-cni-bin\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.336498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336482    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.336672 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336658    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 31 00:02:48.336775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336734    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.336836 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336808    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gvgd9\""
Oct 31 00:02:48.336836 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336824    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.336955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336862    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-9x2zk\""
Oct 31 00:02:48.336955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336887    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.336955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336936    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.336955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336936    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 31 00:02:48.337122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.336916    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 31 00:02:48.337122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337060    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-27h5h\""
Oct 31 00:02:48.337122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337074    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.337122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337087    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-wr2gb\""
Oct 31 00:02:48.337122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337109    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-pp2sk\""
Oct 31 00:02:48.337277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337153    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.337277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337163    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.337688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337675    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf5sc\" (UniqueName: \"kubernetes.io/projected/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-kube-api-access-rf5sc\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.337688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.337683    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zp2t\" (UniqueName: \"kubernetes.io/projected/49220507-cecd-48d6-b19a-3f8479146036-kube-api-access-9zp2t\") pod \"network-check-target-s64vg\" (UID: \"49220507-cecd-48d6-b19a-3f8479146036\") " pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:48.338609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338596    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 31 00:02:48.338778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338757    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.338778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338769    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 31 00:02:48.338778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338772    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a873a458-b1cc-4b84-9a61-a35b5c783868-agent-certs\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:48.338939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338772    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.338939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338798    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 31 00:02:48.338939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.338771    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 31 00:02:48.339270 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.339245    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-7xfz6\""
Oct 31 00:02:48.339397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.339370    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 31 00:02:48.339453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.339438    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9b5br\" (UID: \"b8c6eb6e-5d1c-4f9f-9e44-1dc0645f6fa2\") " pod="openshift-multus/multus-additional-cni-plugins-9b5br"
Oct 31 00:02:48.339689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.339675    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8244x\" (UniqueName: \"kubernetes.io/projected/f14b5c6b-3af4-49b0-a5bb-930d814fa069-kube-api-access-8244x\") pod \"node-resolver-chjbs\" (UID: \"f14b5c6b-3af4-49b0-a5bb-930d814fa069\") " pod="openshift-dns/node-resolver-chjbs"
Oct 31 00:02:48.340588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.340572    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-env-overrides\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.340642 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.340629    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/660c1701-2942-48bd-aee9-ff5341e6ffce-original-pull-secret\") pod \"global-pull-secret-syncer-gfzr6\" (UID: \"660c1701-2942-48bd-aee9-ff5341e6ffce\") " pod="kube-system/global-pull-secret-syncer-gfzr6"
Oct 31 00:02:48.341777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.341748    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a873a458-b1cc-4b84-9a61-a35b5c783868-konnectivity-ca\") pod \"konnectivity-agent-cprhd\" (UID: \"a873a458-b1cc-4b84-9a61-a35b5c783868\") " pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:02:48.341863 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.341847    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 31 00:02:48.341958 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.341945    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.342106 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.342089    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jt67z\""
Oct 31 00:02:48.342482 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.342468    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mddmq\" (UniqueName: \"kubernetes.io/projected/b4e4fdae-7252-4a99-9f43-3939b224757c-kube-api-access-mddmq\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.343435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.343421    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8tq\" (UniqueName: \"kubernetes.io/projected/3ffb287c-6e5e-4adb-9278-b26eec222aeb-kube-api-access-fd8tq\") pod \"aws-ebs-csi-driver-node-4fvq4\" (UID: \"3ffb287c-6e5e-4adb-9278-b26eec222aeb\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-4fvq4"
Oct 31 00:02:48.343891 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.343879    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 31 00:02:48.343934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.343916    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.343971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.343957    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 31 00:02:48.345521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.345502    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gp4rs\" (UniqueName: \"kubernetes.io/projected/19be463a-c645-48ab-a633-42fb816ad71c-kube-api-access-gp4rs\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.345589 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.345576    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.346950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.346936    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 31 00:02:48.347000 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.346951    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvwng\" (UniqueName: \"kubernetes.io/projected/d05191cb-2a10-4929-8000-b87e80b7070f-kube-api-access-gvwng\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.348477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.348449    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phbmw\" (UniqueName: \"kubernetes.io/projected/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-api-access-phbmw\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.348694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.348679    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d05191cb-2a10-4929-8000-b87e80b7070f-serviceca\") pod \"node-ca-f9k85\" (UID: \"d05191cb-2a10-4929-8000-b87e80b7070f\") " pod="openshift-image-registry/node-ca-f9k85"
Oct 31 00:02:48.348724 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.348690    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhzm8\" (UniqueName: \"kubernetes.io/projected/14418f52-35b7-4c4e-9593-cc99bedcb693-kube-api-access-fhzm8\") pod \"service-ca-5fffc97bdf-bkzrc\" (UID: \"14418f52-35b7-4c4e-9593-cc99bedcb693\") " pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc"
Oct 31 00:02:48.348724 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.348666    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/19be463a-c645-48ab-a633-42fb816ad71c-iptables-alerter-script\") pod \"iptables-alerter-9bwpb\" (UID: \"19be463a-c645-48ab-a633-42fb816ad71c\") " pod="openshift-network-operator/iptables-alerter-9bwpb"
Oct 31 00:02:48.348796 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.348785    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/bc469038-9017-4941-972e-0cb0bc3adf6c-multus-daemon-config\") pod \"multus-pqz2l\" (UID: \"bc469038-9017-4941-972e-0cb0bc3adf6c\") " pod="openshift-multus/multus-pqz2l"
Oct 31 00:02:48.349323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.349305    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f9cde72b-3aa1-4c45-95e1-72a55236dc1e-ovnkube-script-lib\") pod \"ovnkube-node-s95vh\" (UID: \"f9cde72b-3aa1-4c45-95e1-72a55236dc1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:48.351415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.351401    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.353802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.353786    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qvx\" (UniqueName: \"kubernetes.io/projected/868e0cf5-15d2-4a00-9fe6-01ff90ac0aae-kube-api-access-z2qvx\") pod \"migrator-66df5d496b-db4f4\" (UID: \"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4"
Oct 31 00:02:48.371244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.370903    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 31 00:02:48.379868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.379845    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") pod \"periodic-gathering-4xtcf-rd6pb\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") " pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:02:48.472374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.472285    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:02:48.837805 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.837772    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:48.837805 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.837811    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.837836    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:48.837862    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.837916    4478 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.837928    4478 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.837940    4478 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.837943    4478 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.837963    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.837952208 +0000 UTC m=+2.854616899 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : configmap "kube-rbac-proxy" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.837984    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.837969862 +0000 UTC m=+2.854634551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.838001    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.83799318 +0000 UTC m=+2.854657870 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : secret "metrics-daemon-secret" not found
Oct 31 00:02:48.838054 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:48.838016    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:49.838007568 +0000 UTC m=+2.854672258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:49.844977 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:49.844934    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:49.844994    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:49.845035    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845069    4478 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845161    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:51.845132063 +0000 UTC m=+4.861796756 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : configmap "kube-rbac-proxy" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:49.845075    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845183    4478 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845232    4478 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845247    4478 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845252    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:51.845235983 +0000 UTC m=+4.861900694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845272    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:51.845260373 +0000 UTC m=+4.861925082 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : secret "metrics-daemon-secret" not found
Oct 31 00:02:49.845474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:49.845306    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:51.845278347 +0000 UTC m=+4.861943061 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:50.171521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:50.171436    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-db4f4" event={"ID":"868e0cf5-15d2-4a00-9fe6-01ff90ac0aae","Type":"ContainerStarted","Data":"7a867933a52141f1811b2b03652a7dad7fc18ded7336f52b4d3f2e13f9273a50"}
Oct 31 00:02:50.172893 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:50.172867    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb" event={"ID":"6f241cd5-d596-4fb1-a270-39d5f3a83727","Type":"ContainerStarted","Data":"e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128"}
Oct 31 00:02:51.860862 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:51.860820    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:51.860941    4478 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:51.860952    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:51.861023    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls podName:b4e4fdae-7252-4a99-9f43-3939b224757c nodeName:}" failed. No retries permitted until 2026-10-31 00:02:55.861001844 +0000 UTC m=+8.877666542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls") pod "dns-default-hlhj6" (UID: "b4e4fdae-7252-4a99-9f43-3939b224757c") : secret "dns-default-metrics-tls" not found
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:51.861078    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:51.861129    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:51.861202    4478 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:02:51.861332 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:51.861256    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:02:55.86124023 +0000 UTC m=+8.877904935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : secret "metrics-daemon-secret" not found
Oct 31 00:02:51.861712 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:51.861570    4478 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:51.861712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:51.861629    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:51.861794 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:51.861722    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:02:55.861676544 +0000 UTC m=+8.878341248 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:52.803208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:52.803142    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:02:52.997596 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:52.997572    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:53.057768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:53.057705    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-chjbs_f14b5c6b-3af4-49b0-a5bb-930d814fa069/dns-node-resolver/0.log"
Oct 31 00:02:53.181547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:53.181516    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-bkzrc" event={"ID":"14418f52-35b7-4c4e-9593-cc99bedcb693","Type":"ContainerStarted","Data":"a675e8a3622117a7fdb2f5e2d24dcdcd4146688f313a9daa414ba49377279ce1"}
Oct 31 00:02:53.859237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:53.859200    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f9k85_d05191cb-2a10-4929-8000-b87e80b7070f/node-ca/0.log"
Oct 31 00:02:54.860652 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:54.860615    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-gjbb9_89ac4cee-c777-4655-9bcc-aa09aa136ab5/serve-healthcheck-canary/0.log"
Oct 31 00:02:54.956312 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:54.956279    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:54.956491 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:54.956480    4478 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 31 00:02:55.030814 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.030792    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-s95vh"
Oct 31 00:02:55.064982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.064962    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-db4f4_868e0cf5-15d2-4a00-9fe6-01ff90ac0aae/migrator/0.log"
Oct 31 00:02:55.888049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.887990    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.888063    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.888095    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:55.888200    4478 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:55.888200    4478 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:55.888273    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs podName:1a6de299-a94e-4591-acac-b5018a133c65 nodeName:}" failed. No retries permitted until 2026-10-31 00:03:03.888254571 +0000 UTC m=+16.904919264 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs") pod "network-metrics-daemon-9cbv5" (UID: "1a6de299-a94e-4591-acac-b5018a133c65") : secret "metrics-daemon-secret" not found
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:02:55.888287    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls podName:c0a001a3-66bc-4816-aa70-7b3c46cbc10f nodeName:}" failed. No retries permitted until 2026-10-31 00:03:03.888281577 +0000 UTC m=+16.904946267 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mvljb" (UID: "c0a001a3-66bc-4816-aa70-7b3c46cbc10f") : secret "insights-runtime-extractor-tls" not found
Oct 31 00:02:55.888489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.888362    4478 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 31 00:02:55.890557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.890532    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/b4e4fdae-7252-4a99-9f43-3939b224757c-metrics-tls\") pod \"dns-default-hlhj6\" (UID: \"b4e4fdae-7252-4a99-9f43-3939b224757c\") " pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:55.970448 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:55.970420    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:02:56.381056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:56.381026    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hlhj6"]
Oct 31 00:02:56.385206 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:02:56.385179    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4e4fdae_7252_4a99_9f43_3939b224757c.slice/crio-f815ed3e82553df64d080ee48a2a245244728a5f7d862fb8176a462d4e3ce27b WatchSource:0}: Error finding container f815ed3e82553df64d080ee48a2a245244728a5f7d862fb8176a462d4e3ce27b: Status 404 returned error can't find the container with id f815ed3e82553df64d080ee48a2a245244728a5f7d862fb8176a462d4e3ce27b
Oct 31 00:02:57.192102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:57.192064    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlhj6" event={"ID":"b4e4fdae-7252-4a99-9f43-3939b224757c","Type":"ContainerStarted","Data":"f815ed3e82553df64d080ee48a2a245244728a5f7d862fb8176a462d4e3ce27b"}
Oct 31 00:02:57.680326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:57.680296    4478 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-137-148.us-west-1.compute.internal" event="NodeReady"
Oct 31 00:02:59.200323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:59.200288    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlhj6" event={"ID":"b4e4fdae-7252-4a99-9f43-3939b224757c","Type":"ContainerStarted","Data":"aa344c192cc061e17cc61899e14205bd4383250217d99cf7bb3eec2ae9ae865b"}
Oct 31 00:02:59.200713 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:59.200348    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hlhj6" event={"ID":"b4e4fdae-7252-4a99-9f43-3939b224757c","Type":"ContainerStarted","Data":"13599930bae7728e6c6de11424ceda8b83c083dd43420ff4401933bacb1bf86d"}
Oct 31 00:02:59.200713 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:02:59.200424    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:03:03.952087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:03.952053    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:03:03.952561 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:03.952098    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:03:03.954555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:03.954521    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1a6de299-a94e-4591-acac-b5018a133c65-metrics-certs\") pod \"network-metrics-daemon-9cbv5\" (UID: \"1a6de299-a94e-4591-acac-b5018a133c65\") " pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:03:03.954664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:03.954604    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0a001a3-66bc-4816-aa70-7b3c46cbc10f-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mvljb\" (UID: \"c0a001a3-66bc-4816-aa70-7b3c46cbc10f\") " pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:03:04.066615 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:04.066586    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9cbv5"
Oct 31 00:03:04.071968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:04.071950    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mvljb"
Oct 31 00:03:04.435435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:04.435405    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9cbv5"]
Oct 31 00:03:04.478195 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:04.478168    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6de299_a94e_4591_acac_b5018a133c65.slice/crio-c3a1a7f5c309083085d158c3b40b33c929fc78a56f710314f5460926cef5d693 WatchSource:0}: Error finding container c3a1a7f5c309083085d158c3b40b33c929fc78a56f710314f5460926cef5d693: Status 404 returned error can't find the container with id c3a1a7f5c309083085d158c3b40b33c929fc78a56f710314f5460926cef5d693
Oct 31 00:03:04.487853 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:04.487830    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mvljb"]
Oct 31 00:03:04.491127 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:04.491106    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0a001a3_66bc_4816_aa70_7b3c46cbc10f.slice/crio-c541b0b163e65b8dae280240f9a412bdd3f1a461e7ebee7c8418940f4d74bf5d WatchSource:0}: Error finding container c541b0b163e65b8dae280240f9a412bdd3f1a461e7ebee7c8418940f4d74bf5d: Status 404 returned error can't find the container with id c541b0b163e65b8dae280240f9a412bdd3f1a461e7ebee7c8418940f4d74bf5d
Oct 31 00:03:05.220526 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:05.220489    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cbv5" event={"ID":"1a6de299-a94e-4591-acac-b5018a133c65","Type":"ContainerStarted","Data":"c3a1a7f5c309083085d158c3b40b33c929fc78a56f710314f5460926cef5d693"}
Oct 31 00:03:05.221954 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:05.221933    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvljb" event={"ID":"c0a001a3-66bc-4816-aa70-7b3c46cbc10f","Type":"ContainerStarted","Data":"472162ed59f1807b2620512cf7954f8f820f171285ba8f3e6325ae072d4ff6c3"}
Oct 31 00:03:05.222063 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:05.221973    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvljb" event={"ID":"c0a001a3-66bc-4816-aa70-7b3c46cbc10f","Type":"ContainerStarted","Data":"c541b0b163e65b8dae280240f9a412bdd3f1a461e7ebee7c8418940f4d74bf5d"}
Oct 31 00:03:06.230094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:06.230045    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cbv5" event={"ID":"1a6de299-a94e-4591-acac-b5018a133c65","Type":"ContainerStarted","Data":"a9e87fe00211e6f4c52014d835e54cb06a5005eff803af4e9a97eadc8966f6f5"}
Oct 31 00:03:06.230562 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:06.230142    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9cbv5" event={"ID":"1a6de299-a94e-4591-acac-b5018a133c65","Type":"ContainerStarted","Data":"4cedb6c3eed398be731c303e545919d89c254c2f21e9758893ba75cdf83641eb"}
Oct 31 00:03:06.231790 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:06.231772    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvljb" event={"ID":"c0a001a3-66bc-4816-aa70-7b3c46cbc10f","Type":"ContainerStarted","Data":"17b0a4138272035c2f69a26e7f170f3e576d2ffb2abb672e8c9654d87659fa23"}
Oct 31 00:03:07.826597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.826520    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5469859dcb-phg8x"]
Oct 31 00:03:07.828249 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.828233    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.935346 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.935304    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 31 00:03:07.982604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982574    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wlq7\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-kube-api-access-5wlq7\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982609    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9ad0c69-6871-493a-8108-0b63f9cf05f7-installation-pull-secrets\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982634    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-registry-tls\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982714    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9ad0c69-6871-493a-8108-0b63f9cf05f7-image-registry-private-configuration\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982768    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9ad0c69-6871-493a-8108-0b63f9cf05f7-registry-certificates\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982836    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-bound-sa-token\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982883    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9ad0c69-6871-493a-8108-0b63f9cf05f7-ca-trust-extracted\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:07.982931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:07.982907    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad0c69-6871-493a-8108-0b63f9cf05f7-trusted-ca\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.029252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.027404    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5469859dcb-phg8x"]
Oct 31 00:03:08.029252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.027635    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 31 00:03:08.034419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.034375    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 31 00:03:08.034893 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.034876    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-z67qm\""
Oct 31 00:03:08.034979 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.034967    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 31 00:03:08.083539 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083481    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9ad0c69-6871-493a-8108-0b63f9cf05f7-installation-pull-secrets\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083539 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083512    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-registry-tls\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083539 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083534    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9ad0c69-6871-493a-8108-0b63f9cf05f7-image-registry-private-configuration\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083553    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9ad0c69-6871-493a-8108-0b63f9cf05f7-registry-certificates\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083581    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-bound-sa-token\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083612    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9ad0c69-6871-493a-8108-0b63f9cf05f7-ca-trust-extracted\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083675    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad0c69-6871-493a-8108-0b63f9cf05f7-trusted-ca\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.083730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.083712    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wlq7\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-kube-api-access-5wlq7\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.084174 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.084148    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9ad0c69-6871-493a-8108-0b63f9cf05f7-ca-trust-extracted\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.084710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.084688    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9ad0c69-6871-493a-8108-0b63f9cf05f7-registry-certificates\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.084960 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.084940    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9ad0c69-6871-493a-8108-0b63f9cf05f7-trusted-ca\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.086203 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.086175    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9ad0c69-6871-493a-8108-0b63f9cf05f7-image-registry-private-configuration\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.086203 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.086185    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-registry-tls\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.086312 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.086218    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9ad0c69-6871-493a-8108-0b63f9cf05f7-installation-pull-secrets\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.144152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.144127    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-bound-sa-token\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.147618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.147595    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wlq7\" (UniqueName: \"kubernetes.io/projected/b9ad0c69-6871-493a-8108-0b63f9cf05f7-kube-api-access-5wlq7\") pod \"image-registry-5469859dcb-phg8x\" (UID: \"b9ad0c69-6871-493a-8108-0b63f9cf05f7\") " pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.240096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.240057    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mvljb" event={"ID":"c0a001a3-66bc-4816-aa70-7b3c46cbc10f","Type":"ContainerStarted","Data":"1737f80f2bd9717f5762024181b0c27f9978c15396db1d1c3c28cd7eee7d6378"}
Oct 31 00:03:08.438396 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.438351    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:08.799686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.799622    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cprhd"
Oct 31 00:03:08.912568 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:08.912541    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5469859dcb-phg8x"]
Oct 31 00:03:08.915614 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:08.915589    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ad0c69_6871_493a_8108_0b63f9cf05f7.slice/crio-f94436438b9f07abf9fa6b46bfee44902a92bb9a4da28d3328431d67f45439ee WatchSource:0}: Error finding container f94436438b9f07abf9fa6b46bfee44902a92bb9a4da28d3328431d67f45439ee: Status 404 returned error can't find the container with id f94436438b9f07abf9fa6b46bfee44902a92bb9a4da28d3328431d67f45439ee
Oct 31 00:03:09.208534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.208505    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hlhj6"
Oct 31 00:03:09.245168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.245133    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5469859dcb-phg8x" event={"ID":"b9ad0c69-6871-493a-8108-0b63f9cf05f7","Type":"ContainerStarted","Data":"46e40cffa6374626acebe5e6e6a0707433f0e0bb274972b441bab0f95efeba33"}
Oct 31 00:03:09.245313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.245181    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5469859dcb-phg8x" event={"ID":"b9ad0c69-6871-493a-8108-0b63f9cf05f7","Type":"ContainerStarted","Data":"f94436438b9f07abf9fa6b46bfee44902a92bb9a4da28d3328431d67f45439ee"}
Oct 31 00:03:09.245313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.245207    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:09.446259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.446205    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5469859dcb-phg8x" podStartSLOduration=2.44619165 podStartE2EDuration="2.44619165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:09.445549609 +0000 UTC m=+22.462214342" watchObservedRunningTime="2026-10-31 00:03:09.44619165 +0000 UTC m=+22.462856405"
Oct 31 00:03:09.842723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.842683    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"]
Oct 31 00:03:09.845506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.845490    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:09.899326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.899296    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mp8kw\""
Oct 31 00:03:09.899479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.899354    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 31 00:03:09.912708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.912682    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"]
Oct 31 00:03:09.996418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:09.996373    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/202b638d-45b7-4353-ae54-24673443627b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f292b\" (UID: \"202b638d-45b7-4353-ae54-24673443627b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:10.096746 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:10.096668    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/202b638d-45b7-4353-ae54-24673443627b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f292b\" (UID: \"202b638d-45b7-4353-ae54-24673443627b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:10.099212 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:10.099189    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/202b638d-45b7-4353-ae54-24673443627b-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-f292b\" (UID: \"202b638d-45b7-4353-ae54-24673443627b\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:10.155185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:10.155158    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:10.521200 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:10.521170    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"]
Oct 31 00:03:10.524725 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:10.524697    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod202b638d_45b7_4353_ae54_24673443627b.slice/crio-5d7aecc46e05e6f2bdf3bc5d4ad6e7827b00f752a15d8502a0786f02a15b2552 WatchSource:0}: Error finding container 5d7aecc46e05e6f2bdf3bc5d4ad6e7827b00f752a15d8502a0786f02a15b2552: Status 404 returned error can't find the container with id 5d7aecc46e05e6f2bdf3bc5d4ad6e7827b00f752a15d8502a0786f02a15b2552
Oct 31 00:03:11.254361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:11.254323    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b" event={"ID":"202b638d-45b7-4353-ae54-24673443627b","Type":"ContainerStarted","Data":"5d7aecc46e05e6f2bdf3bc5d4ad6e7827b00f752a15d8502a0786f02a15b2552"}
Oct 31 00:03:12.259113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:12.259076    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b" event={"ID":"202b638d-45b7-4353-ae54-24673443627b","Type":"ContainerStarted","Data":"edfcdfe1d76b7eda07bf839114b84ac7e2268d2d13ef3c2b0bea78c202cf607d"}
Oct 31 00:03:12.259567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:12.259293    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:12.263949 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:12.263926    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b"
Oct 31 00:03:12.374328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:12.374272    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-f292b" podStartSLOduration=1.982536053 podStartE2EDuration="3.374254767s" totalImagesPullingTime="1.391718714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:12.373674288 +0000 UTC m=+25.390338999" watchObservedRunningTime="2026-10-31 00:03:12.374254767 +0000 UTC m=+25.390919478"
Oct 31 00:03:13.444493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.444460    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"]
Oct 31 00:03:13.446608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.446584    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.518195 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.518159    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.518317 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.518199    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.518317 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.518304    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4pcq\" (UniqueName: \"kubernetes.io/projected/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-kube-api-access-g4pcq\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.518414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.518334    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.548309 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.548280    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Oct 31 00:03:13.548427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.548341    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Oct 31 00:03:13.548427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.548355    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-dwltk\""
Oct 31 00:03:13.548427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.548363    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 31 00:03:13.548574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.548515    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 31 00:03:13.548574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.548536    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 31 00:03:13.598810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.598785    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"]
Oct 31 00:03:13.618564 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.618544    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4pcq\" (UniqueName: \"kubernetes.io/projected/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-kube-api-access-g4pcq\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.618626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.618573    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.618626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.618600    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.618626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.618618    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.619232 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.619207    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.621104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.621081    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.621188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.621138    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.679218 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.679189    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4pcq\" (UniqueName: \"kubernetes.io/projected/9f3c31b2-a6a8-4a46-8f7b-82812f793f38-kube-api-access-g4pcq\") pod \"prometheus-operator-9cd89b74c-wcc94\" (UID: \"9f3c31b2-a6a8-4a46-8f7b-82812f793f38\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:13.757181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:13.757132    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"
Oct 31 00:03:14.126565 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:14.126495    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-wcc94"]
Oct 31 00:03:14.158018 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:14.157990    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3c31b2_a6a8_4a46_8f7b_82812f793f38.slice/crio-40ecf69a8498e1593dc9c88026c3f6a5109c21980fcd6e54b591c21d60a9e207 WatchSource:0}: Error finding container 40ecf69a8498e1593dc9c88026c3f6a5109c21980fcd6e54b591c21d60a9e207: Status 404 returned error can't find the container with id 40ecf69a8498e1593dc9c88026c3f6a5109c21980fcd6e54b591c21d60a9e207
Oct 31 00:03:14.266505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:14.266473    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94" event={"ID":"9f3c31b2-a6a8-4a46-8f7b-82812f793f38","Type":"ContainerStarted","Data":"40ecf69a8498e1593dc9c88026c3f6a5109c21980fcd6e54b591c21d60a9e207"}
Oct 31 00:03:14.755143 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:03:14.755108    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:03:16.275393 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:16.275355    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94" event={"ID":"9f3c31b2-a6a8-4a46-8f7b-82812f793f38","Type":"ContainerStarted","Data":"c0367a06ffbddf646215c735b542588ad07296effb3a26319525e0ff66d4ce4c"}
Oct 31 00:03:16.275836 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:16.275433    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94" event={"ID":"9f3c31b2-a6a8-4a46-8f7b-82812f793f38","Type":"ContainerStarted","Data":"6924ee83118198dbaa71aa8fda5bfbced7944ce78763861a95e1db8c6c6cc774"}
Oct 31 00:03:16.392079 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:16.392022    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-wcc94" podStartSLOduration=1.69873893 podStartE2EDuration="3.392007285s" totalImagesPullingTime="1.693268355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:16.391434962 +0000 UTC m=+29.408099675" watchObservedRunningTime="2026-10-31 00:03:16.392007285 +0000 UTC m=+29.408671998"
Oct 31 00:03:19.186226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.186193    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pp55f"]
Oct 31 00:03:19.219437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.219410    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257170 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257145    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4e5d1-9ced-477b-bd94-b189d25147d7-metrics-client-ca\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257176    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-root\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257199    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257218    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-textfile\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257274    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-tls\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257363    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7jrm\" (UniqueName: \"kubernetes.io/projected/3ff4e5d1-9ced-477b-bd94-b189d25147d7-kube-api-access-k7jrm\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257419    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-sys\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257498    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-wtmp\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.257534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.257524    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.271375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.271352    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sm2z2\""
Oct 31 00:03:19.271375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.271362    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 31 00:03:19.271517 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.271402    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 31 00:03:19.334426 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.334397    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 31 00:03:19.358798 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358767    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-root\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.358920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358811    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.358920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358833    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-textfile\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.358920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358880    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-root\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358921    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-tls\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358945    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7jrm\" (UniqueName: \"kubernetes.io/projected/3ff4e5d1-9ced-477b-bd94-b189d25147d7-kube-api-access-k7jrm\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358962    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-sys\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.358992    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-wtmp\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359018    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359071    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4e5d1-9ced-477b-bd94-b189d25147d7-metrics-client-ca\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359101    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-sys\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359118    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-textfile\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359246    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-wtmp\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359457    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.359720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.359654    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ff4e5d1-9ced-477b-bd94-b189d25147d7-metrics-client-ca\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.361493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.361470    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.361626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.361477    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/3ff4e5d1-9ced-477b-bd94-b189d25147d7-node-exporter-tls\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.419805 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.419774    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7jrm\" (UniqueName: \"kubernetes.io/projected/3ff4e5d1-9ced-477b-bd94-b189d25147d7-kube-api-access-k7jrm\") pod \"node-exporter-pp55f\" (UID: \"3ff4e5d1-9ced-477b-bd94-b189d25147d7\") " pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.570087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:19.570016    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pp55f"
Oct 31 00:03:19.603873 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:19.603843    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff4e5d1_9ced_477b_bd94_b189d25147d7.slice/crio-8df58fbae9c3591ce17560366b28a7c5e39c983a1d81934cee76c764af7c983f WatchSource:0}: Error finding container 8df58fbae9c3591ce17560366b28a7c5e39c983a1d81934cee76c764af7c983f: Status 404 returned error can't find the container with id 8df58fbae9c3591ce17560366b28a7c5e39c983a1d81934cee76c764af7c983f
Oct 31 00:03:20.292920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:20.292886    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp55f" event={"ID":"3ff4e5d1-9ced-477b-bd94-b189d25147d7","Type":"ContainerStarted","Data":"8df58fbae9c3591ce17560366b28a7c5e39c983a1d81934cee76c764af7c983f"}
Oct 31 00:03:22.305799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:22.305764    4478 generic.go:356] "Generic (PLEG): container finished" podID="3ff4e5d1-9ced-477b-bd94-b189d25147d7" containerID="18fadaea76a37056461030ddc2cf2acc7475bd804a69f2f33cca06ed5dbddf8a" exitCode=0
Oct 31 00:03:22.305799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:22.305801    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp55f" event={"ID":"3ff4e5d1-9ced-477b-bd94-b189d25147d7","Type":"ContainerDied","Data":"18fadaea76a37056461030ddc2cf2acc7475bd804a69f2f33cca06ed5dbddf8a"}
Oct 31 00:03:22.806244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:22.806211    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-s64vg"
Oct 31 00:03:23.312799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:23.312765    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp55f" event={"ID":"3ff4e5d1-9ced-477b-bd94-b189d25147d7","Type":"ContainerStarted","Data":"7a8e8a1dcc538f316fcff13541a4fcee65fb05f14f6ce191048e6fa0e88930f9"}
Oct 31 00:03:23.313244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:23.312828    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pp55f" event={"ID":"3ff4e5d1-9ced-477b-bd94-b189d25147d7","Type":"ContainerStarted","Data":"974b267051d6699a2964f0afe0be130cad4b8550b230219818f3675c51634ffd"}
Oct 31 00:03:23.432270 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:23.432208    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pp55f" podStartSLOduration=2.29861637 podStartE2EDuration="4.432194419s" totalImagesPullingTime="2.045174027s" totalInitContainerRuntime="88.404022ms" isStatefulPod=true podCreationTimestamp="2026-10-31 00:03:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:23.430970914 +0000 UTC m=+36.447635638" watchObservedRunningTime="2026-10-31 00:03:23.432194419 +0000 UTC m=+36.448859131"
Oct 31 00:03:30.253294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:30.253214    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5469859dcb-phg8x"
Oct 31 00:03:35.648992 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.648961    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-7779b64d85-qjqv7"]
Oct 31 00:03:35.651098 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.651078    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687367    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzrh8\" (UniqueName: \"kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687416    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687441    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687545    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687591    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687733 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687673    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.687733 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.687704    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.755047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755020    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-vxxnz\""
Oct 31 00:03:35.755047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755043    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 31 00:03:35.755226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755028    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 31 00:03:35.755226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755020    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 31 00:03:35.755226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755090    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 31 00:03:35.755226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755135    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 31 00:03:35.755226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755177    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 31 00:03:35.755467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.755454    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 31 00:03:35.788948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.788923    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.788954    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.788985    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzrh8\" (UniqueName: \"kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789007    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789037    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789067    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789097    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789766 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789707    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789788    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.789920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.789808    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.790123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.790039    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.791683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.791665    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.791773 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.791733    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.805259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.805239    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7779b64d85-qjqv7"]
Oct 31 00:03:35.850049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.850027    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzrh8\" (UniqueName: \"kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8\") pod \"console-7779b64d85-qjqv7\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") " pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:35.960998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:35.960970    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:36.341786 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:36.341710    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7779b64d85-qjqv7"]
Oct 31 00:03:36.373632 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:36.373598    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod413447bc_7d42_4ef9_a9e4_07f80cfd305f.slice/crio-c8c3be3e87e9bf5d6e2c29ec237637e8e2eac557fa00747acc7e442b4960ca66 WatchSource:0}: Error finding container c8c3be3e87e9bf5d6e2c29ec237637e8e2eac557fa00747acc7e442b4960ca66: Status 404 returned error can't find the container with id c8c3be3e87e9bf5d6e2c29ec237637e8e2eac557fa00747acc7e442b4960ca66
Oct 31 00:03:37.359787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:37.359747    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7779b64d85-qjqv7" event={"ID":"413447bc-7d42-4ef9-a9e4-07f80cfd305f","Type":"ContainerStarted","Data":"c8c3be3e87e9bf5d6e2c29ec237637e8e2eac557fa00747acc7e442b4960ca66"}
Oct 31 00:03:40.372035 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:40.371957    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7779b64d85-qjqv7" event={"ID":"413447bc-7d42-4ef9-a9e4-07f80cfd305f","Type":"ContainerStarted","Data":"28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26"}
Oct 31 00:03:40.491710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:40.491660    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-7779b64d85-qjqv7" podStartSLOduration=1.8171200509999998 podStartE2EDuration="5.491646359s" totalImagesPullingTime="3.674526308s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:40.490744333 +0000 UTC m=+53.507409045" watchObservedRunningTime="2026-10-31 00:03:40.491646359 +0000 UTC m=+53.508311070"
Oct 31 00:03:44.756104 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:03:44.756059    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:03:45.961396 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:45.961349    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:45.961817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:45.961419    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:45.966090 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:45.966068    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:46.394935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:46.394863    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:03:48.071248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.071221    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-78468b6dcf-wsj6l"]
Oct 31 00:03:48.075767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.075750    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.176757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.176730    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzpm7\" (UniqueName: \"kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.176902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.176764    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.176902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.176786    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.177024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.176953    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.177024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.177007    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.177110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.177085    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.177158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.177146    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.177192 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.177179    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.277806 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277784    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.277921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277817    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzpm7\" (UniqueName: \"kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.277921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277835    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.277921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277853    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.277921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277882    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.277921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277914    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.278165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277938    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.278165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.277964    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.278302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.278282    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.278593 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.278572    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.278689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.278650    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.278689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.278657    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.280346 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.280325    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 31 00:03:48.280592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.280567    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.280775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.280755    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.281337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.281320    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78468b6dcf-wsj6l"]
Oct 31 00:03:48.289234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.289209    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.338557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.338504    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzpm7\" (UniqueName: \"kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7\") pod \"console-78468b6dcf-wsj6l\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") " pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.386948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.386925    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:48.758178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:48.758151    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-78468b6dcf-wsj6l"]
Oct 31 00:03:48.763204 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:03:48.763177    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba94eab_de51_43fd_b139_47979ddadd8f.slice/crio-00ba67383649b88513c3e5b37d1373dd0c6f401f7f2984e3c2848ad3210ec972 WatchSource:0}: Error finding container 00ba67383649b88513c3e5b37d1373dd0c6f401f7f2984e3c2848ad3210ec972: Status 404 returned error can't find the container with id 00ba67383649b88513c3e5b37d1373dd0c6f401f7f2984e3c2848ad3210ec972
Oct 31 00:03:49.401639 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:49.401607    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78468b6dcf-wsj6l" event={"ID":"0ba94eab-de51-43fd-b139-47979ddadd8f","Type":"ContainerStarted","Data":"a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea"}
Oct 31 00:03:49.402026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:49.401653    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78468b6dcf-wsj6l" event={"ID":"0ba94eab-de51-43fd-b139-47979ddadd8f","Type":"ContainerStarted","Data":"00ba67383649b88513c3e5b37d1373dd0c6f401f7f2984e3c2848ad3210ec972"}
Oct 31 00:03:49.520670 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:49.520610    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-78468b6dcf-wsj6l" podStartSLOduration=1.520597389 podStartE2EDuration="1.520597389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:03:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:03:49.520114927 +0000 UTC m=+62.536779638" watchObservedRunningTime="2026-10-31 00:03:49.520597389 +0000 UTC m=+62.537262093"
Oct 31 00:03:58.387993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:58.387963    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:58.388376 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:58.388043    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:58.392840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:58.392819    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:58.432567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:58.432546    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:03:58.659228 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:03:58.659151    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7779b64d85-qjqv7"]
Oct 31 00:04:03.940588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:03.940554    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74bdfdb997-gw7rx"]
Oct 31 00:04:03.944416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:03.944398    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.046069 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.046042    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bdfdb997-gw7rx"]
Oct 31 00:04:04.095907 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.095880    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-oauth-serving-cert\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096061 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.095914    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a580ef3c-c479-4597-b217-504205006944-console-serving-cert\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096061 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.095935    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a580ef3c-c479-4597-b217-504205006944-tmp\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.096065    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-trusted-ca-bundle\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.096089    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqj2s\" (UniqueName: \"kubernetes.io/projected/a580ef3c-c479-4597-b217-504205006944-kube-api-access-xqj2s\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.096115    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a580ef3c-c479-4597-b217-504205006944-console-oauth-config\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096222 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.096177    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-console-config\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.096254 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.096226    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-service-ca\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.157355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.157324    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4xtcf-rd6pb"]
Oct 31 00:04:04.157535 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.157514    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb" podUID="6f241cd5-d596-4fb1-a270-39d5f3a83727" containerName="insights-gathering" containerID="cri-o://e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128" gracePeriod=30
Oct 31 00:04:04.197279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197222    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a580ef3c-c479-4597-b217-504205006944-console-oauth-config\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197261    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-console-config\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197282    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-service-ca\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197314    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-oauth-serving-cert\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197584 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197565    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a580ef3c-c479-4597-b217-504205006944-console-serving-cert\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197599    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a580ef3c-c479-4597-b217-504205006944-tmp\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197644    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-trusted-ca-bundle\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.197751 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197682    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqj2s\" (UniqueName: \"kubernetes.io/projected/a580ef3c-c479-4597-b217-504205006944-kube-api-access-xqj2s\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.198029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.197999    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-console-config\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.198109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.198028    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a580ef3c-c479-4597-b217-504205006944-tmp\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.198109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.198033    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-oauth-serving-cert\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.198109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.198067    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-service-ca\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.198843 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.198810    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a580ef3c-c479-4597-b217-504205006944-trusted-ca-bundle\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.199974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.199952    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a580ef3c-c479-4597-b217-504205006944-console-oauth-config\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.200089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.200072    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a580ef3c-c479-4597-b217-504205006944-console-serving-cert\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.258680 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.258658    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqj2s\" (UniqueName: \"kubernetes.io/projected/a580ef3c-c479-4597-b217-504205006944-kube-api-access-xqj2s\") pod \"console-74bdfdb997-gw7rx\" (UID: \"a580ef3c-c479-4597-b217-504205006944\") " pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.397427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.397405    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:04:04.450635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.450565    4478 generic.go:356] "Generic (PLEG): container finished" podID="6f241cd5-d596-4fb1-a270-39d5f3a83727" containerID="e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128" exitCode=2
Oct 31 00:04:04.450635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.450613    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb"
Oct 31 00:04:04.450793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.450643    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb" event={"ID":"6f241cd5-d596-4fb1-a270-39d5f3a83727","Type":"ContainerDied","Data":"e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128"}
Oct 31 00:04:04.450793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.450688    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-4xtcf-rd6pb" event={"ID":"6f241cd5-d596-4fb1-a270-39d5f3a83727","Type":"ContainerDied","Data":"07385e3cdb25aa1134dc26c37e8cb63bf971b96bd4aea27481084673a624c871"}
Oct 31 00:04:04.450793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.450710    4478 scope.go:122] "RemoveContainer" containerID="e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128"
Oct 31 00:04:04.458522 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.458503    4478 scope.go:122] "RemoveContainer" containerID="e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128"
Oct 31 00:04:04.458831 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:04:04.458805    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128\": container with ID starting with e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128 not found: ID does not exist" containerID="e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128"
Oct 31 00:04:04.458908 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.458842    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128"} err="failed to get container status \"e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128\": rpc error: code = NotFound desc = could not find container \"e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128\": container with ID starting with e61fc269eb5878a729283eac1ecea8e2b56fb12b238164aee14fcd0c83896128 not found: ID does not exist"
Oct 31 00:04:04.501414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501374    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") pod \"6f241cd5-d596-4fb1-a270-39d5f3a83727\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") "
Oct 31 00:04:04.501513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501460    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") pod \"6f241cd5-d596-4fb1-a270-39d5f3a83727\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") "
Oct 31 00:04:04.501513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501500    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") pod \"6f241cd5-d596-4fb1-a270-39d5f3a83727\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") "
Oct 31 00:04:04.501622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501532    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") pod \"6f241cd5-d596-4fb1-a270-39d5f3a83727\" (UID: \"6f241cd5-d596-4fb1-a270-39d5f3a83727\") "
Oct 31 00:04:04.501778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501757    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path" pod "6f241cd5-d596-4fb1-a270-39d5f3a83727" (UID: "6f241cd5-d596-4fb1-a270-39d5f3a83727"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:04:04.501822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501772    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle" pod "6f241cd5-d596-4fb1-a270-39d5f3a83727" (UID: "6f241cd5-d596-4fb1-a270-39d5f3a83727"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:04.501822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.501783    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle" pod "6f241cd5-d596-4fb1-a270-39d5f3a83727" (UID: "6f241cd5-d596-4fb1-a270-39d5f3a83727"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:04.503872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.503851    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6" pod "6f241cd5-d596-4fb1-a270-39d5f3a83727" (UID: "6f241cd5-d596-4fb1-a270-39d5f3a83727"). InnerVolumeSpecName "kube-api-access-zbjn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:04:04.557075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.557050    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:04.602401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.602355    4478 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-trusted-ca-bundle\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:04.602401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.602401    4478 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/6f241cd5-d596-4fb1-a270-39d5f3a83727-archives-path\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:04.602550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.602414    4478 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6f241cd5-d596-4fb1-a270-39d5f3a83727-service-ca-bundle\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:04.602550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.602423    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbjn6\" (UniqueName: \"kubernetes.io/projected/6f241cd5-d596-4fb1-a270-39d5f3a83727-kube-api-access-zbjn6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:04.869306 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.869271    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-4xtcf-rd6pb"]
Oct 31 00:04:04.871835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.871812    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-4xtcf-rd6pb"]
Oct 31 00:04:04.979911 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:04.979878    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74bdfdb997-gw7rx"]
Oct 31 00:04:04.983762 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:04:04.983732    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda580ef3c_c479_4597_b217_504205006944.slice/crio-6066755c957f4c598526100d5a7b6363bc003429f37fe7cfe9e9e81b9a53bd4e WatchSource:0}: Error finding container 6066755c957f4c598526100d5a7b6363bc003429f37fe7cfe9e9e81b9a53bd4e: Status 404 returned error can't find the container with id 6066755c957f4c598526100d5a7b6363bc003429f37fe7cfe9e9e81b9a53bd4e
Oct 31 00:04:05.114579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:05.114548    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f241cd5-d596-4fb1-a270-39d5f3a83727" path="/var/lib/kubelet/pods/6f241cd5-d596-4fb1-a270-39d5f3a83727/volumes"
Oct 31 00:04:05.455448 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:05.455371    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bdfdb997-gw7rx" event={"ID":"a580ef3c-c479-4597-b217-504205006944","Type":"ContainerStarted","Data":"8518dd7245322be660e1b8be0b36865c8ae9c9bf3c889a2883687d60e8ded986"}
Oct 31 00:04:05.455636 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:05.455472    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74bdfdb997-gw7rx" event={"ID":"a580ef3c-c479-4597-b217-504205006944","Type":"ContainerStarted","Data":"6066755c957f4c598526100d5a7b6363bc003429f37fe7cfe9e9e81b9a53bd4e"}
Oct 31 00:04:05.574098 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:05.574040    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74bdfdb997-gw7rx" podStartSLOduration=2.574025543 podStartE2EDuration="2.574025543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:04:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:04:05.572921421 +0000 UTC m=+78.589586150" watchObservedRunningTime="2026-10-31 00:04:05.574025543 +0000 UTC m=+78.590690252"
Oct 31 00:04:14.557188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:14.557149    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:14.557696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:14.557215    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:14.567236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:14.567211    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:14.756022 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:04:14.755988    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:04:15.497966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:15.497935    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74bdfdb997-gw7rx"
Oct 31 00:04:15.666271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:15.666238    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78468b6dcf-wsj6l"]
Oct 31 00:04:23.680118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.680061    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-7779b64d85-qjqv7" podUID="413447bc-7d42-4ef9-a9e4-07f80cfd305f" containerName="console" containerID="cri-o://28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26" gracePeriod=15
Oct 31 00:04:23.924679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.924657    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:04:23.952279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952230    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952311    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952365    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8\" (UniqueName: \"kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952414    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952454    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952502    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952538    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp\") pod \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\" (UID: \"413447bc-7d42-4ef9-a9e4-07f80cfd305f\") "
Oct 31 00:04:23.952743 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952640    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:23.952902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952833    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:23.952902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952840    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:23.953168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.952935    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:04:23.954732 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.954707    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "kube-api-access-xzrh8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:04:23.954815 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.954735    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:23.954815 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:23.954770    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config" pod "413447bc-7d42-4ef9-a9e4-07f80cfd305f" (UID: "413447bc-7d42-4ef9-a9e4-07f80cfd305f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:24.053589 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053564    4478 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-config\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.053589 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053586    4478 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-oauth-config\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.053736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053596    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzrh8\" (UniqueName: \"kubernetes.io/projected/413447bc-7d42-4ef9-a9e4-07f80cfd305f-kube-api-access-xzrh8\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.053736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053605    4478 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.053736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053613    4478 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/413447bc-7d42-4ef9-a9e4-07f80cfd305f-oauth-serving-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.053736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053622    4478 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/413447bc-7d42-4ef9-a9e4-07f80cfd305f-console-serving-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.053736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.053631    4478 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/413447bc-7d42-4ef9-a9e4-07f80cfd305f-tmp\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:24.524913 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.524879    4478 generic.go:356] "Generic (PLEG): container finished" podID="413447bc-7d42-4ef9-a9e4-07f80cfd305f" containerID="28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26" exitCode=0
Oct 31 00:04:24.525112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.524932    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7779b64d85-qjqv7"
Oct 31 00:04:24.525112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.524954    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7779b64d85-qjqv7" event={"ID":"413447bc-7d42-4ef9-a9e4-07f80cfd305f","Type":"ContainerDied","Data":"28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26"}
Oct 31 00:04:24.525112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.524994    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7779b64d85-qjqv7" event={"ID":"413447bc-7d42-4ef9-a9e4-07f80cfd305f","Type":"ContainerDied","Data":"c8c3be3e87e9bf5d6e2c29ec237637e8e2eac557fa00747acc7e442b4960ca66"}
Oct 31 00:04:24.525112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.525012    4478 scope.go:122] "RemoveContainer" containerID="28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26"
Oct 31 00:04:24.532948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.532933    4478 scope.go:122] "RemoveContainer" containerID="28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26"
Oct 31 00:04:24.533238 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:04:24.533219    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26\": container with ID starting with 28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26 not found: ID does not exist" containerID="28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26"
Oct 31 00:04:24.533299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.533251    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26"} err="failed to get container status \"28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26\": rpc error: code = NotFound desc = could not find container \"28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26\": container with ID starting with 28d564a27e19b2eb0878c0221c0164fef338e881e984b6742c89b02e0542ca26 not found: ID does not exist"
Oct 31 00:04:24.688892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.688865    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7779b64d85-qjqv7"]
Oct 31 00:04:24.738741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:24.738715    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7779b64d85-qjqv7"]
Oct 31 00:04:25.113983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:25.113949    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="413447bc-7d42-4ef9-a9e4-07f80cfd305f" path="/var/lib/kubelet/pods/413447bc-7d42-4ef9-a9e4-07f80cfd305f/volumes"
Oct 31 00:04:40.685815 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.685764    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-78468b6dcf-wsj6l" podUID="0ba94eab-de51-43fd-b139-47979ddadd8f" containerName="console" containerID="cri-o://a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea" gracePeriod=15
Oct 31 00:04:40.935716 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.935692    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:04:40.964895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.964874    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.964993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.964920    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.964993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.964938    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.964993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.964956    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.965153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.964993    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.965153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965012    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.965153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965040    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7\" (UniqueName: \"kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.965153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965088    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp\") pod \"0ba94eab-de51-43fd-b139-47979ddadd8f\" (UID: \"0ba94eab-de51-43fd-b139-47979ddadd8f\") "
Oct 31 00:04:40.965457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965310    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:40.965573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965464    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:04:40.965573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965539    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:40.965691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965563    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:40.965691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.965619    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:04:40.967183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.967157    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:40.967259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.967229    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:04:40.967555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:40.967530    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7" pod "0ba94eab-de51-43fd-b139-47979ddadd8f" (UID: "0ba94eab-de51-43fd-b139-47979ddadd8f"). InnerVolumeSpecName "kube-api-access-fzpm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:04:41.066183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066156    4478 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-oauth-serving-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066188    4478 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-serving-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066204    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzpm7\" (UniqueName: \"kubernetes.io/projected/0ba94eab-de51-43fd-b139-47979ddadd8f-kube-api-access-fzpm7\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066221    4478 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ba94eab-de51-43fd-b139-47979ddadd8f-tmp\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066236    4478 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-console-config\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066249    4478 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-trusted-ca-bundle\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066264    4478 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0ba94eab-de51-43fd-b139-47979ddadd8f-console-oauth-config\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.066289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.066278    4478 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba94eab-de51-43fd-b139-47979ddadd8f-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:04:41.585741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.585707    4478 generic.go:356] "Generic (PLEG): container finished" podID="0ba94eab-de51-43fd-b139-47979ddadd8f" containerID="a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea" exitCode=0
Oct 31 00:04:41.585897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.585762    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-78468b6dcf-wsj6l"
Oct 31 00:04:41.585897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.585761    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78468b6dcf-wsj6l" event={"ID":"0ba94eab-de51-43fd-b139-47979ddadd8f","Type":"ContainerDied","Data":"a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea"}
Oct 31 00:04:41.585966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.585905    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-78468b6dcf-wsj6l" event={"ID":"0ba94eab-de51-43fd-b139-47979ddadd8f","Type":"ContainerDied","Data":"00ba67383649b88513c3e5b37d1373dd0c6f401f7f2984e3c2848ad3210ec972"}
Oct 31 00:04:41.585966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.585936    4478 scope.go:122] "RemoveContainer" containerID="a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea"
Oct 31 00:04:41.593580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.593563    4478 scope.go:122] "RemoveContainer" containerID="a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea"
Oct 31 00:04:41.593803 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:04:41.593784    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea\": container with ID starting with a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea not found: ID does not exist" containerID="a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea"
Oct 31 00:04:41.593879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.593830    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea"} err="failed to get container status \"a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea\": rpc error: code = NotFound desc = could not find container \"a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea\": container with ID starting with a96e514b391ff5b21bc15e2d6ff54e472925aebd2adeba9df7af74430b71feea not found: ID does not exist"
Oct 31 00:04:41.748369 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.748344    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-78468b6dcf-wsj6l"]
Oct 31 00:04:41.797966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:41.797936    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-78468b6dcf-wsj6l"]
Oct 31 00:04:43.113996 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:04:43.113967    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ba94eab-de51-43fd-b139-47979ddadd8f" path="/var/lib/kubelet/pods/0ba94eab-de51-43fd-b139-47979ddadd8f/volumes"
Oct 31 00:04:44.755591 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:04:44.755558    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:05:14.755180 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:05:14.755106    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:05:44.755026 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:05:44.754990    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:06:14.755317 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:06:14.755283    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:06:44.755897 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:06:44.755858    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:07:14.755704 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:07:14.755666    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:07:44.755506 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:07:44.755468    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:08:14.754821 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:08:14.754735    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:08:44.755590 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:08:44.755548    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:09:14.755332 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:09:14.755295    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:09:15.871598 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871565    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"]
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871812    4478 cpu_manager.go:414] "containerMap: removing container" podUID="0ba94eab-de51-43fd-b139-47979ddadd8f" containerName="console"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871825    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ba94eab-de51-43fd-b139-47979ddadd8f" containerName="console"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871862    4478 cpu_manager.go:414] "containerMap: removing container" podUID="6f241cd5-d596-4fb1-a270-39d5f3a83727" containerName="insights-gathering"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871870    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f241cd5-d596-4fb1-a270-39d5f3a83727" containerName="insights-gathering"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871892    4478 cpu_manager.go:414] "containerMap: removing container" podUID="413447bc-7d42-4ef9-a9e4-07f80cfd305f" containerName="console"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871900    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="413447bc-7d42-4ef9-a9e4-07f80cfd305f" containerName="console"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871943    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f241cd5-d596-4fb1-a270-39d5f3a83727" containerName="insights-gathering"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871952    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="413447bc-7d42-4ef9-a9e4-07f80cfd305f" containerName="console"
Oct 31 00:09:15.872066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.871958    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ba94eab-de51-43fd-b139-47979ddadd8f" containerName="console"
Oct 31 00:09:15.874641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.874626    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.943672 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.943652    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 31 00:09:15.943776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.943754    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 31 00:09:15.943945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.943933    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 31 00:09:15.990340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990318    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.990458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990342    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.990458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990362    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.990458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990397    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.990587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990512    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.990587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990535    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.990587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.990562    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7c47\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:15.991990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.991976    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 31 00:09:15.991990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.991982    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 31 00:09:15.992100 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.992004    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 31 00:09:15.992100 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:15.992009    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-g5g6j\""
Oct 31 00:09:16.018301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.018274    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"]
Oct 31 00:09:16.091143 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091118    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091147    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091172    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7c47\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091200    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091321    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091368    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091417    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.091834 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.091805    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.093599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.093573    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.093839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.093818    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.093893 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.093837    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.093947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.093928    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.167091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.167037    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7c47\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.181029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.181007    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-psjjr\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.185916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.185901    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:16.611802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.611781    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"]
Oct 31 00:09:16.614310 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:09:16.614277    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0adf63_c8e2_4ccc_bfd9_72118b0763fb.slice/crio-fc4f548a4a0f6fa39f555b3dff886b85592cdd97d863671f8f4115edfa2e2f57 WatchSource:0}: Error finding container fc4f548a4a0f6fa39f555b3dff886b85592cdd97d863671f8f4115edfa2e2f57: Status 404 returned error can't find the container with id fc4f548a4a0f6fa39f555b3dff886b85592cdd97d863671f8f4115edfa2e2f57
Oct 31 00:09:16.616050 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:16.616036    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:09:17.435796 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:17.435762    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" event={"ID":"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb","Type":"ContainerStarted","Data":"fc4f548a4a0f6fa39f555b3dff886b85592cdd97d863671f8f4115edfa2e2f57"}
Oct 31 00:09:20.521133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:20.521106    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:09:20.521394 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:20.521168    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:09:21.455274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:21.455242    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" event={"ID":"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb","Type":"ContainerStarted","Data":"d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd"}
Oct 31 00:09:21.455500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:21.455480    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:21.457120 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:21.457080    4478 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-psjjr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:09:21.457237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:21.457143    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:09:21.662779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:21.662731    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" podStartSLOduration=2.758549338 podStartE2EDuration="6.662719659s" totalImagesPullingTime="3.904170321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:21.662237271 +0000 UTC m=+394.678901983" watchObservedRunningTime="2026-10-31 00:09:21.662719659 +0000 UTC m=+394.679384370"
Oct 31 00:09:22.458908 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:22.458871    4478 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-psjjr container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:09:22.459076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:22.458934    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:09:25.459550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:09:25.459522    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:09:44.755959 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:09:44.755924    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:10:05.639878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.639849    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"]
Oct 31 00:10:05.642902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.642885    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.701727 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.701701    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 31 00:10:05.702151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.702135    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 31 00:10:05.734068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.734042    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.734187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.734070    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.734187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.734096    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.734187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.734156    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b869\" (UniqueName: \"kubernetes.io/projected/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-kube-api-access-2b869\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.734339 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.734317    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.734418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.734401    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.751434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.751399    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 31 00:10:05.751556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.751437    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-25dgf\""
Oct 31 00:10:05.751556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.751405    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 31 00:10:05.751556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.751493    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 31 00:10:05.752920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.752903    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"]
Oct 31 00:10:05.834917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.834886    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.835084 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.834923    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b869\" (UniqueName: \"kubernetes.io/projected/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-kube-api-access-2b869\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.835084 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.835064    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.835205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.835139    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.835205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.835198    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.835313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.835228    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.835547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.835527    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-manifests\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.837528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.837503    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-tmp\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.837616 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.837531    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-webhook-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.837716 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.837696    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-metrics-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.837752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.837695    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.895611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.895560    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b869\" (UniqueName: \"kubernetes.io/projected/8f82e628-1bd5-4173-9fcd-13e3b5b81c97-kube-api-access-2b869\") pod \"opendatahub-operator-controller-manager-8646f9bf7f-vj88j\" (UID: \"8f82e628-1bd5-4173-9fcd-13e3b5b81c97\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:05.956579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:05.956548    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:06.547948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:06.547919    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"]
Oct 31 00:10:06.551517 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:10:06.551494    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f82e628_1bd5_4173_9fcd_13e3b5b81c97.slice/crio-9c7c348d82caa0615d7bf303d0d4605524093dbfa16437d26380706cab410b90 WatchSource:0}: Error finding container 9c7c348d82caa0615d7bf303d0d4605524093dbfa16437d26380706cab410b90: Status 404 returned error can't find the container with id 9c7c348d82caa0615d7bf303d0d4605524093dbfa16437d26380706cab410b90
Oct 31 00:10:06.595816 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:06.595777    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j" event={"ID":"8f82e628-1bd5-4173-9fcd-13e3b5b81c97","Type":"ContainerStarted","Data":"9c7c348d82caa0615d7bf303d0d4605524093dbfa16437d26380706cab410b90"}
Oct 31 00:10:10.612045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:10.611950    4478 generic.go:356] "Generic (PLEG): container finished" podID="8f82e628-1bd5-4173-9fcd-13e3b5b81c97" containerID="440d5e03ad15c9448bff28dc390dc31dafa8471d593ba80de067c660b21f0e37" exitCode=0
Oct 31 00:10:10.612045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:10.611994    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j" event={"ID":"8f82e628-1bd5-4173-9fcd-13e3b5b81c97","Type":"ContainerDied","Data":"440d5e03ad15c9448bff28dc390dc31dafa8471d593ba80de067c660b21f0e37"}
Oct 31 00:10:11.617637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:11.617607    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j" event={"ID":"8f82e628-1bd5-4173-9fcd-13e3b5b81c97","Type":"ContainerStarted","Data":"f9a9403e33930651811ca0146f182ef311b2bacde21dc7f2d23afecda70a118c"}
Oct 31 00:10:11.618014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:11.617722    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:11.866571 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:11.866517    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j" podStartSLOduration=2.330817245 podStartE2EDuration="6.866502374s" totalImagesPullingTime="4.062867377s" totalInitContainerRuntime="472.817752ms" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:11.817119735 +0000 UTC m=+444.833784482" watchObservedRunningTime="2026-10-31 00:10:11.866502374 +0000 UTC m=+444.883167085"
Oct 31 00:10:14.755185 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:10:14.755147    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:10:22.623928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:22.623896    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-8646f9bf7f-vj88j"
Oct 31 00:10:39.055043 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.055011    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"]
Oct 31 00:10:39.059720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.059701    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.081418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.081371    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr9ln\" (UniqueName: \"kubernetes.io/projected/a68e8a72-f823-45d3-ad92-0c597c6433d3-kube-api-access-fr9ln\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.081418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.081417    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a68e8a72-f823-45d3-ad92-0c597c6433d3-tmp\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.081581 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.081531    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68e8a72-f823-45d3-ad92-0c597c6433d3-tls-certs\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.165490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.165454    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"]
Oct 31 00:10:39.182066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.182041    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68e8a72-f823-45d3-ad92-0c597c6433d3-tls-certs\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.182204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.182090    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr9ln\" (UniqueName: \"kubernetes.io/projected/a68e8a72-f823-45d3-ad92-0c597c6433d3-kube-api-access-fr9ln\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.182204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.182135    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a68e8a72-f823-45d3-ad92-0c597c6433d3-tmp\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.184465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.184429    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a68e8a72-f823-45d3-ad92-0c597c6433d3-tmp\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.187929 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.187913    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 31 00:10:39.227300 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.227276    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mk2zv\""
Oct 31 00:10:39.227300 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.227295    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 31 00:10:39.237896 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.237870    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a68e8a72-f823-45d3-ad92-0c597c6433d3-tls-certs\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.242492 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.242468    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr9ln\" (UniqueName: \"kubernetes.io/projected/a68e8a72-f823-45d3-ad92-0c597c6433d3-kube-api-access-fr9ln\") pod \"kube-auth-proxy-5cb564c48d-fslfx\" (UID: \"a68e8a72-f823-45d3-ad92-0c597c6433d3\") " pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.370089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.370011    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"
Oct 31 00:10:39.828742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:39.828709    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx"]
Oct 31 00:10:39.831149 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:10:39.831126    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda68e8a72_f823_45d3_ad92_0c597c6433d3.slice/crio-4f6e6f323e9302bf214d0e41c408a274078002aecd3120ada9951a3db5c3c318 WatchSource:0}: Error finding container 4f6e6f323e9302bf214d0e41c408a274078002aecd3120ada9951a3db5c3c318: Status 404 returned error can't find the container with id 4f6e6f323e9302bf214d0e41c408a274078002aecd3120ada9951a3db5c3c318
Oct 31 00:10:40.717764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:40.717717    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx" event={"ID":"a68e8a72-f823-45d3-ad92-0c597c6433d3","Type":"ContainerStarted","Data":"4f6e6f323e9302bf214d0e41c408a274078002aecd3120ada9951a3db5c3c318"}
Oct 31 00:10:43.729681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:43.729644    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx" event={"ID":"a68e8a72-f823-45d3-ad92-0c597c6433d3","Type":"ContainerStarted","Data":"a422ca2ffa6a7cc6c8c9434322596619f3d6b77091b9757d9885d5d3d17c37ff"}
Oct 31 00:10:43.845802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:43.845747    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb564c48d-fslfx" podStartSLOduration=1.721349496 podStartE2EDuration="4.845732151s" totalImagesPullingTime="3.124382655s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:43.844834843 +0000 UTC m=+476.861499574" watchObservedRunningTime="2026-10-31 00:10:43.845732151 +0000 UTC m=+476.862396897"
Oct 31 00:10:44.755479 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:10:44.755444    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:10:48.033713 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.033682    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp9zx"]
Oct 31 00:10:48.038153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.038131    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:10:48.092254 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.092232    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 31 00:10:48.092610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.092587    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 31 00:10:48.093278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.093260    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp9zx"]
Oct 31 00:10:48.141882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.141859    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7p7cn\""
Oct 31 00:10:48.160030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.160005    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2j7g\" (UniqueName: \"kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g\") pod \"authorino-56c6c9ddd5-cp9zx\" (UID: \"afcc8091-a183-435d-be4d-a38aa7507fc0\") " pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:10:48.260587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.260558    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2j7g\" (UniqueName: \"kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g\") pod \"authorino-56c6c9ddd5-cp9zx\" (UID: \"afcc8091-a183-435d-be4d-a38aa7507fc0\") " pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:10:48.321450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.321368    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2j7g\" (UniqueName: \"kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g\") pod \"authorino-56c6c9ddd5-cp9zx\" (UID: \"afcc8091-a183-435d-be4d-a38aa7507fc0\") " pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:10:48.360279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.360252    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:10:48.729302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.729263    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp9zx"]
Oct 31 00:10:48.736858 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:10:48.736825    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafcc8091_a183_435d_be4d_a38aa7507fc0.slice/crio-93a5e85e895b1772e8224b3ea1f7e59fd84daab2614ece4a9c4d54b0a18b9c39 WatchSource:0}: Error finding container 93a5e85e895b1772e8224b3ea1f7e59fd84daab2614ece4a9c4d54b0a18b9c39: Status 404 returned error can't find the container with id 93a5e85e895b1772e8224b3ea1f7e59fd84daab2614ece4a9c4d54b0a18b9c39
Oct 31 00:10:48.746221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.746197    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx" event={"ID":"afcc8091-a183-435d-be4d-a38aa7507fc0","Type":"ContainerStarted","Data":"93a5e85e895b1772e8224b3ea1f7e59fd84daab2614ece4a9c4d54b0a18b9c39"}
Oct 31 00:10:48.844825 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.844793    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"]
Oct 31 00:10:48.849123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.849107    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:48.902679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.902660    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m8lzx\""
Oct 31 00:10:48.909777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.909753    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"]
Oct 31 00:10:48.966672 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.966641    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90600f9-ec16-4ae5-af38-f60501b5ffb6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zbvhk\" (UID: \"a90600f9-ec16-4ae5-af38-f60501b5ffb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:48.966772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:48.966748    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7zsd\" (UniqueName: \"kubernetes.io/projected/a90600f9-ec16-4ae5-af38-f60501b5ffb6-kube-api-access-d7zsd\") pod \"kuadrant-operator-controller-manager-678d87766f-zbvhk\" (UID: \"a90600f9-ec16-4ae5-af38-f60501b5ffb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:49.067136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.067068    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7zsd\" (UniqueName: \"kubernetes.io/projected/a90600f9-ec16-4ae5-af38-f60501b5ffb6-kube-api-access-d7zsd\") pod \"kuadrant-operator-controller-manager-678d87766f-zbvhk\" (UID: \"a90600f9-ec16-4ae5-af38-f60501b5ffb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:49.067136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.067109    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90600f9-ec16-4ae5-af38-f60501b5ffb6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zbvhk\" (UID: \"a90600f9-ec16-4ae5-af38-f60501b5ffb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:49.067532 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.067451    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a90600f9-ec16-4ae5-af38-f60501b5ffb6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-zbvhk\" (UID: \"a90600f9-ec16-4ae5-af38-f60501b5ffb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:49.137221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.137195    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7zsd\" (UniqueName: \"kubernetes.io/projected/a90600f9-ec16-4ae5-af38-f60501b5ffb6-kube-api-access-d7zsd\") pod \"kuadrant-operator-controller-manager-678d87766f-zbvhk\" (UID: \"a90600f9-ec16-4ae5-af38-f60501b5ffb6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:49.160888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.160853    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:49.636401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.636351    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"]
Oct 31 00:10:49.640573 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:10:49.640533    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90600f9_ec16_4ae5_af38_f60501b5ffb6.slice/crio-c6b8fc822f75b55bec6f1cd8c44f4888c8c6a4fb15cd83973fbe247d876e04b7 WatchSource:0}: Error finding container c6b8fc822f75b55bec6f1cd8c44f4888c8c6a4fb15cd83973fbe247d876e04b7: Status 404 returned error can't find the container with id c6b8fc822f75b55bec6f1cd8c44f4888c8c6a4fb15cd83973fbe247d876e04b7
Oct 31 00:10:49.750986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:49.750948    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk" event={"ID":"a90600f9-ec16-4ae5-af38-f60501b5ffb6","Type":"ContainerStarted","Data":"c6b8fc822f75b55bec6f1cd8c44f4888c8c6a4fb15cd83973fbe247d876e04b7"}
Oct 31 00:10:52.766827 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:52.766764    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx" event={"ID":"afcc8091-a183-435d-be4d-a38aa7507fc0","Type":"ContainerStarted","Data":"a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad"}
Oct 31 00:10:52.886578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:52.886516    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx" podStartSLOduration=2.339073845 podStartE2EDuration="5.886500817s" totalImagesPullingTime="3.547426972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:52.884033805 +0000 UTC m=+485.900698522" watchObservedRunningTime="2026-10-31 00:10:52.886500817 +0000 UTC m=+485.903165535"
Oct 31 00:10:54.777242 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:54.777163    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk" event={"ID":"a90600f9-ec16-4ae5-af38-f60501b5ffb6","Type":"ContainerStarted","Data":"c85e151c0c0262de0974c8abcffad47b012b45e31cf9fe8b9cb3edc9f58195bf"}
Oct 31 00:10:54.777658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:54.777312    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:10:55.001964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:10:54.998457    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk" podStartSLOduration=2.212949636 podStartE2EDuration="6.998431768s" totalImagesPullingTime="4.785482132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:54.947216414 +0000 UTC m=+487.963881143" watchObservedRunningTime="2026-10-31 00:10:54.998431768 +0000 UTC m=+488.015096479"
Oct 31 00:11:05.782766 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:05.782692    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-zbvhk"
Oct 31 00:11:14.755991 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:11:14.755954    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:11:30.554574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.554539    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-57c568df98-jc6t5"]
Oct 31 00:11:30.557052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.557033    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.609828 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.609805    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Oct 31 00:11:30.640372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.640334    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Oct 31 00:11:30.659023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.659002    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 00:11:30.659023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.659006    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 00:11:30.659238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.659224    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-zq9pr\""
Oct 31 00:11:30.659279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.659243    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Oct 31 00:11:30.660325 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.660310    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c568df98-jc6t5"]
Oct 31 00:11:30.693437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.693408    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.693545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.693451    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q2bd\" (UniqueName: \"kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.693545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.693480    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.794850 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.794824    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q2bd\" (UniqueName: \"kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.794985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.794856    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.794985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.794906    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.797571 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.797545    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.797684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.797587    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.855850 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.855796    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q2bd\" (UniqueName: \"kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd\") pod \"maas-controller-57c568df98-jc6t5\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") " pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:30.870677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:30.870655    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:31.243059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:31.243031    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-57c568df98-jc6t5"]
Oct 31 00:11:31.277252 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:11:31.277226    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod157259e6_b715_4187_955a_d95ea4184b4d.slice/crio-f85be743f2e6e8632a60b98f929c51468d2cd0ef9db4b8d06102a155d2c59792 WatchSource:0}: Error finding container f85be743f2e6e8632a60b98f929c51468d2cd0ef9db4b8d06102a155d2c59792: Status 404 returned error can't find the container with id f85be743f2e6e8632a60b98f929c51468d2cd0ef9db4b8d06102a155d2c59792
Oct 31 00:11:31.908439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:31.908406    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c568df98-jc6t5" event={"ID":"157259e6-b715-4187-955a-d95ea4184b4d","Type":"ContainerStarted","Data":"f85be743f2e6e8632a60b98f929c51468d2cd0ef9db4b8d06102a155d2c59792"}
Oct 31 00:11:43.953569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:43.953531    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c568df98-jc6t5" event={"ID":"157259e6-b715-4187-955a-d95ea4184b4d","Type":"ContainerStarted","Data":"dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46"}
Oct 31 00:11:43.953950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:43.953727    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:44.073076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:44.073025    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-57c568df98-jc6t5" podStartSLOduration=2.049787892 podStartE2EDuration="14.073010517s" totalImagesPullingTime="12.023222625s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:44.072770531 +0000 UTC m=+537.089435289" watchObservedRunningTime="2026-10-31 00:11:44.073010517 +0000 UTC m=+537.089675230"
Oct 31 00:11:44.756412 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:11:44.756360    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:11:54.959528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:54.959500    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:55.341235 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.341163    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57c568df98-jc6t5"]
Oct 31 00:11:55.341401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.341344    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-57c568df98-jc6t5" podUID="157259e6-b715-4187-955a-d95ea4184b4d" containerName="manager" containerID="cri-o://dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46" gracePeriod=30
Oct 31 00:11:55.575692 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.575670    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:55.682838 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.682813    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert\") pod \"157259e6-b715-4187-955a-d95ea4184b4d\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") "
Oct 31 00:11:55.682971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.682858    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd\" (UniqueName: \"kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd\") pod \"157259e6-b715-4187-955a-d95ea4184b4d\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") "
Oct 31 00:11:55.682971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.682903    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs\") pod \"157259e6-b715-4187-955a-d95ea4184b4d\" (UID: \"157259e6-b715-4187-955a-d95ea4184b4d\") "
Oct 31 00:11:55.685147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.685117    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert" pod "157259e6-b715-4187-955a-d95ea4184b4d" (UID: "157259e6-b715-4187-955a-d95ea4184b4d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:11:55.685147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.685133    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs" pod "157259e6-b715-4187-955a-d95ea4184b4d" (UID: "157259e6-b715-4187-955a-d95ea4184b4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:11:55.685285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.685119    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd" pod "157259e6-b715-4187-955a-d95ea4184b4d" (UID: "157259e6-b715-4187-955a-d95ea4184b4d"). InnerVolumeSpecName "kube-api-access-8q2bd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:11:55.783477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.783452    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:55.783477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.783474    4478 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/157259e6-b715-4187-955a-d95ea4184b4d-webhook-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:55.783653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:55.783486    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q2bd\" (UniqueName: \"kubernetes.io/projected/157259e6-b715-4187-955a-d95ea4184b4d-kube-api-access-8q2bd\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:11:56.004594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.004498    4478 generic.go:356] "Generic (PLEG): container finished" podID="157259e6-b715-4187-955a-d95ea4184b4d" containerID="dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46" exitCode=0
Oct 31 00:11:56.004594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.004567    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c568df98-jc6t5" event={"ID":"157259e6-b715-4187-955a-d95ea4184b4d","Type":"ContainerDied","Data":"dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46"}
Oct 31 00:11:56.004594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.004585    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-57c568df98-jc6t5"
Oct 31 00:11:56.005199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.004647    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-57c568df98-jc6t5" event={"ID":"157259e6-b715-4187-955a-d95ea4184b4d","Type":"ContainerDied","Data":"f85be743f2e6e8632a60b98f929c51468d2cd0ef9db4b8d06102a155d2c59792"}
Oct 31 00:11:56.005199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.004674    4478 scope.go:122] "RemoveContainer" containerID="dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46"
Oct 31 00:11:56.013625 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.013607    4478 scope.go:122] "RemoveContainer" containerID="dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46"
Oct 31 00:11:56.013889 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:11:56.013872    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46\": container with ID starting with dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46 not found: ID does not exist" containerID="dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46"
Oct 31 00:11:56.013948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.013897    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46"} err="failed to get container status \"dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46\": rpc error: code = NotFound desc = could not find container \"dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46\": container with ID starting with dabf55fd90e0a4ac27c6d881f1e1d3a9e4ef5e0e12e1eb3e915b924e057f2a46 not found: ID does not exist"
Oct 31 00:11:56.168603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.168566    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-57c568df98-jc6t5"]
Oct 31 00:11:56.217993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:56.217954    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-57c568df98-jc6t5"]
Oct 31 00:11:57.115003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:11:57.114971    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="157259e6-b715-4187-955a-d95ea4184b4d" path="/var/lib/kubelet/pods/157259e6-b715-4187-955a-d95ea4184b4d/volumes"
Oct 31 00:12:06.921712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.921675    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"]
Oct 31 00:12:06.922117 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.921950    4478 cpu_manager.go:414] "containerMap: removing container" podUID="157259e6-b715-4187-955a-d95ea4184b4d" containerName="manager"
Oct 31 00:12:06.922117 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.921961    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="157259e6-b715-4187-955a-d95ea4184b4d" containerName="manager"
Oct 31 00:12:06.922117 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.922023    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="157259e6-b715-4187-955a-d95ea4184b4d" containerName="manager"
Oct 31 00:12:06.923742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.923727    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991132    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1abf08-492e-4e29-a821-6378d706ab91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991161    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce1abf08-492e-4e29-a821-6378d706ab91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991349 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991185    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991349 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991320    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991441 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991369    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzwsg\" (UniqueName: \"kubernetes.io/projected/ce1abf08-492e-4e29-a821-6378d706ab91-kube-api-access-hzwsg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991484 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991464    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1abf08-492e-4e29-a821-6378d706ab91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:06.991549 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:06.991535    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.029830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.029801    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"]
Oct 31 00:12:07.092020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.091991    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.092186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092027    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzwsg\" (UniqueName: \"kubernetes.io/projected/ce1abf08-492e-4e29-a821-6378d706ab91-kube-api-access-hzwsg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.092186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092073    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1abf08-492e-4e29-a821-6378d706ab91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.092186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092107    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.092328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092250    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1abf08-492e-4e29-a821-6378d706ab91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.092328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092287    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce1abf08-492e-4e29-a821-6378d706ab91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.092328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092319    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.093012 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.092986    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ce1abf08-492e-4e29-a821-6378d706ab91-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.094634 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.094612    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce1abf08-492e-4e29-a821-6378d706ab91-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.094781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.094654    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.095103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.095086    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.095165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.095145    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ce1abf08-492e-4e29-a821-6378d706ab91-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.153168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.153137    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce1abf08-492e-4e29-a821-6378d706ab91-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.153168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.153162    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzwsg\" (UniqueName: \"kubernetes.io/projected/ce1abf08-492e-4e29-a821-6378d706ab91-kube-api-access-hzwsg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-r842r\" (UID: \"ce1abf08-492e-4e29-a821-6378d706ab91\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.234272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.234241    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:07.685769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.685738    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"]
Oct 31 00:12:07.688767 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:12:07.688739    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1abf08_492e_4e29_a821_6378d706ab91.slice/crio-2e72ad5edb4d49a450e34e1286d9a506b866652b9fb150135579898519c93a67 WatchSource:0}: Error finding container 2e72ad5edb4d49a450e34e1286d9a506b866652b9fb150135579898519c93a67: Status 404 returned error can't find the container with id 2e72ad5edb4d49a450e34e1286d9a506b866652b9fb150135579898519c93a67
Oct 31 00:12:07.690808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.690768    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:12:07.690891 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:07.690834    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:12:08.050372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:08.050287    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r" event={"ID":"ce1abf08-492e-4e29-a821-6378d706ab91","Type":"ContainerStarted","Data":"2562558fd3a8d04c20a617bdd73646fdd1e499fb82ff5a5a45eb0f2bdca5f37a"}
Oct 31 00:12:08.050372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:08.050351    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r" event={"ID":"ce1abf08-492e-4e29-a821-6378d706ab91","Type":"ContainerStarted","Data":"2e72ad5edb4d49a450e34e1286d9a506b866652b9fb150135579898519c93a67"}
Oct 31 00:12:08.050817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:08.050479    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:08.052051 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:08.052030    4478 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r842r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:12:08.052153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:08.052074    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r" podUID="ce1abf08-492e-4e29-a821-6378d706ab91" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:12:08.171291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:08.171219    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r" podStartSLOduration=2.171203442 podStartE2EDuration="2.171203442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:08.170911777 +0000 UTC m=+561.187576515" watchObservedRunningTime="2026-10-31 00:12:08.171203442 +0000 UTC m=+561.187868152"
Oct 31 00:12:09.055055 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:09.055025    4478 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-r842r container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:12:09.055453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:09.055071    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r" podUID="ce1abf08-492e-4e29-a821-6378d706ab91" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:12:12.055976 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.055940    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-r842r"
Oct 31 00:12:12.249079 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.249041    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"]
Oct 31 00:12:12.249355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.249327    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerName="discovery" containerID="cri-o://d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd" gracePeriod=30
Oct 31 00:12:12.488811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.488789    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:12:12.637758 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637688    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.637758 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637738    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.637985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637769    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.637985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637823    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.637985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637880    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.637985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637914    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.637985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.637950    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts\") pod \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\" (UID: \"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb\") "
Oct 31 00:12:12.638370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.638337    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:12:12.640405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.640349    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:12.640524 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.640479    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "kube-api-access-k7c47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:12.640592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.640580    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:12:12.640732 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.640705    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:12.640732 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.640717    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:12.640856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.640757    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts" pod "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" (UID: "6d0adf63-c8e2-4ccc-bfd9-72118b0763fb"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:12.679869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.679848    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wtds2"]
Oct 31 00:12:12.680135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.680123    4478 cpu_manager.go:414] "containerMap: removing container" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerName="discovery"
Oct 31 00:12:12.680181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.680135    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerName="discovery"
Oct 31 00:12:12.680226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.680185    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerName="discovery"
Oct 31 00:12:12.681967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.681943    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:12.736195 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.736170    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 31 00:12:12.736303 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.736283    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 31 00:12:12.736303 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.736286    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-9pvd7\""
Oct 31 00:12:12.736418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.736354    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 31 00:12:12.738737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738713    4478 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-dns-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.738857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738741    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7c47\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-kube-api-access-k7c47\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.738857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738758    4478 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-cacerts\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.738857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738772    4478 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-local-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.738857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738785    4478 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-kubeconfig\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.738857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738808    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.738857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.738822    4478 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb-istio-csr-ca-configmap\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:12.739935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.739911    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wtds2"]
Oct 31 00:12:12.839632 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.839600    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3df50a6f-b1a2-41eb-ae05-a032cca1f2c3-cert\") pod \"kserve-controller-manager-776cc66d47-wtds2\" (UID: \"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:12.839757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.839634    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zzzs\" (UniqueName: \"kubernetes.io/projected/3df50a6f-b1a2-41eb-ae05-a032cca1f2c3-kube-api-access-8zzzs\") pod \"kserve-controller-manager-776cc66d47-wtds2\" (UID: \"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:12.940719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.940688    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3df50a6f-b1a2-41eb-ae05-a032cca1f2c3-cert\") pod \"kserve-controller-manager-776cc66d47-wtds2\" (UID: \"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:12.940847 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.940731    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zzzs\" (UniqueName: \"kubernetes.io/projected/3df50a6f-b1a2-41eb-ae05-a032cca1f2c3-kube-api-access-8zzzs\") pod \"kserve-controller-manager-776cc66d47-wtds2\" (UID: \"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:12.943239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:12.943211    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3df50a6f-b1a2-41eb-ae05-a032cca1f2c3-cert\") pod \"kserve-controller-manager-776cc66d47-wtds2\" (UID: \"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:13.001165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.001142    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zzzs\" (UniqueName: \"kubernetes.io/projected/3df50a6f-b1a2-41eb-ae05-a032cca1f2c3-kube-api-access-8zzzs\") pod \"kserve-controller-manager-776cc66d47-wtds2\" (UID: \"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3\") " pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:13.072005 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.071972    4478 generic.go:356] "Generic (PLEG): container finished" podID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" containerID="d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd" exitCode=0
Oct 31 00:12:13.072419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.072018    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"
Oct 31 00:12:13.072419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.072024    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" event={"ID":"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb","Type":"ContainerDied","Data":"d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd"}
Oct 31 00:12:13.072419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.072064    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr" event={"ID":"6d0adf63-c8e2-4ccc-bfd9-72118b0763fb","Type":"ContainerDied","Data":"fc4f548a4a0f6fa39f555b3dff886b85592cdd97d863671f8f4115edfa2e2f57"}
Oct 31 00:12:13.072419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.072082    4478 scope.go:122] "RemoveContainer" containerID="d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd"
Oct 31 00:12:13.081128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.081104    4478 scope.go:122] "RemoveContainer" containerID="d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd"
Oct 31 00:12:13.081469 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:12:13.081451    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd\": container with ID starting with d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd not found: ID does not exist" containerID="d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd"
Oct 31 00:12:13.081540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.081481    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd"} err="failed to get container status \"d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd\": rpc error: code = NotFound desc = could not find container \"d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd\": container with ID starting with d5fe1f46c441d08af5aa624d75bf325cc5da9f2bc3f8be1ce43140fea96833bd not found: ID does not exist"
Oct 31 00:12:13.235413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.235324    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"]
Oct 31 00:12:13.238997 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.238974    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-psjjr"]
Oct 31 00:12:13.292957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.292935    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:13.661947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:13.661874    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-wtds2"]
Oct 31 00:12:13.665867 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:12:13.665840    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3df50a6f_b1a2_41eb_ae05_a032cca1f2c3.slice/crio-a1609fca363996d81a6763d1b85309033c64f4827ecbd668eaf2c3bcbbde2092 WatchSource:0}: Error finding container a1609fca363996d81a6763d1b85309033c64f4827ecbd668eaf2c3bcbbde2092: Status 404 returned error can't find the container with id a1609fca363996d81a6763d1b85309033c64f4827ecbd668eaf2c3bcbbde2092
Oct 31 00:12:14.078031 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:14.077997    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2" event={"ID":"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3","Type":"ContainerStarted","Data":"a1609fca363996d81a6763d1b85309033c64f4827ecbd668eaf2c3bcbbde2092"}
Oct 31 00:12:14.758870 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:12:14.758761    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:12:15.115354 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:15.115285    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0adf63-c8e2-4ccc-bfd9-72118b0763fb" path="/var/lib/kubelet/pods/6d0adf63-c8e2-4ccc-bfd9-72118b0763fb/volumes"
Oct 31 00:12:17.091281 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:17.091245    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2" event={"ID":"3df50a6f-b1a2-41eb-ae05-a032cca1f2c3","Type":"ContainerStarted","Data":"5ae00adb996eb7df80cecb718e7780577acf7022f9867eca267fbcb94b32774f"}
Oct 31 00:12:17.091779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:17.091324    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:12:17.231531 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:17.231471    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2" podStartSLOduration=2.6010866139999997 podStartE2EDuration="5.231457676s" totalImagesPullingTime="2.630371062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:17.230275515 +0000 UTC m=+570.246940228" watchObservedRunningTime="2026-10-31 00:12:17.231457676 +0000 UTC m=+570.248122388"
Oct 31 00:12:18.639729 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.639698    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"]
Oct 31 00:12:18.645178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.645158    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.697422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.697397    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 31 00:12:18.700077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.700063    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 31 00:12:18.747196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.747172    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 31 00:12:18.750866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.750836    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"]
Oct 31 00:12:18.784269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.784249    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Oct 31 00:12:18.789575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.789553    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.789645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.789581    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.789645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.789598    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.789645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.789613    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.796722 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.796703    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-442zl\""
Oct 31 00:12:18.890590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.890528    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.890590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.890556    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.890590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.890572    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.890590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.890588    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.891147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.891127    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.893232 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.893205    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.893321 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.893215    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.951008 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.950982    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn\") pod \"maas-api-5d95c86d89-nzwtw\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") " pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:18.957830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:18.957812    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:19.337370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:19.337337    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"]
Oct 31 00:12:19.341226 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:12:19.341202    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fa7173_f13c_4036_93c3_c6f048c6c5f4.slice/crio-208bb4ff933b42a5e9774dda24d40a664187472f77c93fb55cfe4d1da87ea65d WatchSource:0}: Error finding container 208bb4ff933b42a5e9774dda24d40a664187472f77c93fb55cfe4d1da87ea65d: Status 404 returned error can't find the container with id 208bb4ff933b42a5e9774dda24d40a664187472f77c93fb55cfe4d1da87ea65d
Oct 31 00:12:20.111443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:20.111409    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw" event={"ID":"89fa7173-f13c-4036-93c3-c6f048c6c5f4","Type":"ContainerStarted","Data":"208bb4ff933b42a5e9774dda24d40a664187472f77c93fb55cfe4d1da87ea65d"}
Oct 31 00:12:25.131421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:25.131357    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw" event={"ID":"89fa7173-f13c-4036-93c3-c6f048c6c5f4","Type":"ContainerStarted","Data":"c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe"}
Oct 31 00:12:25.131784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:25.131511    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:25.255743 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:25.255690    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw" podStartSLOduration=1.564765522 podStartE2EDuration="7.255676622s" totalImagesPullingTime="5.6909111s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:25.255042187 +0000 UTC m=+578.271706917" watchObservedRunningTime="2026-10-31 00:12:25.255676622 +0000 UTC m=+578.272341335"
Oct 31 00:12:31.140592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:31.140524    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:32.221195 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.221159    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"]
Oct 31 00:12:32.224453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.224436    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.295113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.295078    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab82044d-5a0c-4638-a676-5ec869d85173-maas-api-tls\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.295261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.295126    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab82044d-5a0c-4638-a676-5ec869d85173-metrics-certs\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.295318 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.295285    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbvpt\" (UniqueName: \"kubernetes.io/projected/ab82044d-5a0c-4638-a676-5ec869d85173-kube-api-access-gbvpt\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.295355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.295323    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab82044d-5a0c-4638-a676-5ec869d85173-openshift-service-ca\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.326951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.326904    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"]
Oct 31 00:12:32.396022 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.395993    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab82044d-5a0c-4638-a676-5ec869d85173-maas-api-tls\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.396159 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.396028    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab82044d-5a0c-4638-a676-5ec869d85173-metrics-certs\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.396159 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.396094    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbvpt\" (UniqueName: \"kubernetes.io/projected/ab82044d-5a0c-4638-a676-5ec869d85173-kube-api-access-gbvpt\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.396159 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.396113    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab82044d-5a0c-4638-a676-5ec869d85173-openshift-service-ca\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.396935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.396817    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab82044d-5a0c-4638-a676-5ec869d85173-openshift-service-ca\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.398897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.398873    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab82044d-5a0c-4638-a676-5ec869d85173-maas-api-tls\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.398988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.398873    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab82044d-5a0c-4638-a676-5ec869d85173-metrics-certs\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.457060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.457028    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbvpt\" (UniqueName: \"kubernetes.io/projected/ab82044d-5a0c-4638-a676-5ec869d85173-kube-api-access-gbvpt\") pod \"maas-api-78c8449567-7vh5p\" (UID: \"ab82044d-5a0c-4638-a676-5ec869d85173\") " pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.536704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.536644    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:32.908100 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:32.908032    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"]
Oct 31 00:12:32.911225 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:12:32.911197    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab82044d_5a0c_4638_a676_5ec869d85173.slice/crio-1740ae95ab0db226e2d4772454ab5401a2632e276396e3d209be55d471997f8d WatchSource:0}: Error finding container 1740ae95ab0db226e2d4772454ab5401a2632e276396e3d209be55d471997f8d: Status 404 returned error can't find the container with id 1740ae95ab0db226e2d4772454ab5401a2632e276396e3d209be55d471997f8d
Oct 31 00:12:33.161328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:33.161241    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p" event={"ID":"ab82044d-5a0c-4638-a676-5ec869d85173","Type":"ContainerStarted","Data":"1740ae95ab0db226e2d4772454ab5401a2632e276396e3d209be55d471997f8d"}
Oct 31 00:12:34.166658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:34.166586    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p" event={"ID":"ab82044d-5a0c-4638-a676-5ec869d85173","Type":"ContainerStarted","Data":"c7a87862bb9f1855d28c4a97db23ae44919a7e9357deca2627d6b3e1d8a90470"}
Oct 31 00:12:34.167021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:34.166659    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:34.287772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:34.287720    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p" podStartSLOduration=1.463856178 podStartE2EDuration="2.287706963s" totalImagesPullingTime="823.850785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:34.286411324 +0000 UTC m=+587.303076044" watchObservedRunningTime="2026-10-31 00:12:34.287706963 +0000 UTC m=+587.304371713"
Oct 31 00:12:35.694072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:35.694043    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp9zx"]
Oct 31 00:12:35.694455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:35.694211    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx" podUID="afcc8091-a183-435d-be4d-a38aa7507fc0" containerName="authorino" containerID="cri-o://a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad" gracePeriod=30
Oct 31 00:12:35.946582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:35.946525    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:12:36.025946 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.025923    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g\" (UniqueName: \"kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g\") pod \"afcc8091-a183-435d-be4d-a38aa7507fc0\" (UID: \"afcc8091-a183-435d-be4d-a38aa7507fc0\") "
Oct 31 00:12:36.028146 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.028120    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g" pod "afcc8091-a183-435d-be4d-a38aa7507fc0" (UID: "afcc8091-a183-435d-be4d-a38aa7507fc0"). InnerVolumeSpecName "kube-api-access-v2j7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:36.127080 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.127052    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2j7g\" (UniqueName: \"kubernetes.io/projected/afcc8091-a183-435d-be4d-a38aa7507fc0-kube-api-access-v2j7g\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:36.174240 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.174207    4478 generic.go:356] "Generic (PLEG): container finished" podID="afcc8091-a183-435d-be4d-a38aa7507fc0" containerID="a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad" exitCode=0
Oct 31 00:12:36.174419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.174256    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx"
Oct 31 00:12:36.174419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.174261    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx" event={"ID":"afcc8091-a183-435d-be4d-a38aa7507fc0","Type":"ContainerDied","Data":"a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad"}
Oct 31 00:12:36.174419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.174293    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp9zx" event={"ID":"afcc8091-a183-435d-be4d-a38aa7507fc0","Type":"ContainerDied","Data":"93a5e85e895b1772e8224b3ea1f7e59fd84daab2614ece4a9c4d54b0a18b9c39"}
Oct 31 00:12:36.174419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.174313    4478 scope.go:122] "RemoveContainer" containerID="a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad"
Oct 31 00:12:36.183134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.183118    4478 scope.go:122] "RemoveContainer" containerID="a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad"
Oct 31 00:12:36.183370 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:12:36.183349    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad\": container with ID starting with a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad not found: ID does not exist" containerID="a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad"
Oct 31 00:12:36.183438 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.183395    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad"} err="failed to get container status \"a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad\": rpc error: code = NotFound desc = could not find container \"a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad\": container with ID starting with a78617b2db079ade41676d011f34d3efaa1b8265dd6be96b16e0a9085c7f0aad not found: ID does not exist"
Oct 31 00:12:36.344266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.344205    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp9zx"]
Oct 31 00:12:36.346995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:36.346972    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp9zx"]
Oct 31 00:12:37.115139 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:37.115110    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afcc8091-a183-435d-be4d-a38aa7507fc0" path="/var/lib/kubelet/pods/afcc8091-a183-435d-be4d-a38aa7507fc0/volumes"
Oct 31 00:12:40.175420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.175370    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78c8449567-7vh5p"
Oct 31 00:12:40.343575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.343541    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"]
Oct 31 00:12:40.343816 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.343788    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw" podUID="89fa7173-f13c-4036-93c3-c6f048c6c5f4" containerName="maas-api" containerID="cri-o://c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe" gracePeriod=30
Oct 31 00:12:40.582791 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.582769    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:40.662085 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.662061    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca\") pod \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") "
Oct 31 00:12:40.662208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.662123    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs\") pod \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") "
Oct 31 00:12:40.662208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.662160    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls\") pod \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") "
Oct 31 00:12:40.662208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.662202    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn\") pod \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\" (UID: \"89fa7173-f13c-4036-93c3-c6f048c6c5f4\") "
Oct 31 00:12:40.662498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.662474    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca" pod "89fa7173-f13c-4036-93c3-c6f048c6c5f4" (UID: "89fa7173-f13c-4036-93c3-c6f048c6c5f4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:12:40.664393 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.664355    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn" pod "89fa7173-f13c-4036-93c3-c6f048c6c5f4" (UID: "89fa7173-f13c-4036-93c3-c6f048c6c5f4"). InnerVolumeSpecName "kube-api-access-szwpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:12:40.664489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.664425    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls" pod "89fa7173-f13c-4036-93c3-c6f048c6c5f4" (UID: "89fa7173-f13c-4036-93c3-c6f048c6c5f4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:40.664489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.664463    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs" pod "89fa7173-f13c-4036-93c3-c6f048c6c5f4" (UID: "89fa7173-f13c-4036-93c3-c6f048c6c5f4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:12:40.762975 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.762907    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:40.762975 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.762931    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89fa7173-f13c-4036-93c3-c6f048c6c5f4-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:40.762975 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.762940    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szwpn\" (UniqueName: \"kubernetes.io/projected/89fa7173-f13c-4036-93c3-c6f048c6c5f4-kube-api-access-szwpn\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:40.762975 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:40.762951    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89fa7173-f13c-4036-93c3-c6f048c6c5f4-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:12:41.194326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.194296    4478 generic.go:356] "Generic (PLEG): container finished" podID="89fa7173-f13c-4036-93c3-c6f048c6c5f4" containerID="c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe" exitCode=0
Oct 31 00:12:41.194720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.194340    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"
Oct 31 00:12:41.194720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.194350    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw" event={"ID":"89fa7173-f13c-4036-93c3-c6f048c6c5f4","Type":"ContainerDied","Data":"c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe"}
Oct 31 00:12:41.194720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.194409    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw" event={"ID":"89fa7173-f13c-4036-93c3-c6f048c6c5f4","Type":"ContainerDied","Data":"208bb4ff933b42a5e9774dda24d40a664187472f77c93fb55cfe4d1da87ea65d"}
Oct 31 00:12:41.194720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.194430    4478 scope.go:122] "RemoveContainer" containerID="c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe"
Oct 31 00:12:41.202355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.202339    4478 scope.go:122] "RemoveContainer" containerID="c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe"
Oct 31 00:12:41.202639 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:12:41.202614    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe\": container with ID starting with c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe not found: ID does not exist" containerID="c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe"
Oct 31 00:12:41.202701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.202648    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe"} err="failed to get container status \"c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe\": rpc error: code = NotFound desc = could not find container \"c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe\": container with ID starting with c6629ac940e7968350611666b6d5faf8f1ffea05a9d11eab0a38d63cfeaa25fe not found: ID does not exist"
Oct 31 00:12:41.397766 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.397694    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"]
Oct 31 00:12:41.446993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:41.446967    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d95c86d89-nzwtw"]
Oct 31 00:12:43.114659 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:43.114629    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fa7173-f13c-4036-93c3-c6f048c6c5f4" path="/var/lib/kubelet/pods/89fa7173-f13c-4036-93c3-c6f048c6c5f4/volumes"
Oct 31 00:12:44.756275 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:12:44.756239    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:12:48.100762 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:12:48.100731    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-wtds2"
Oct 31 00:13:14.755230 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:13:14.755192    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:13:44.755683 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:13:44.755641    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:14:12.116058 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.115972    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-txds2"]
Oct 31 00:14:12.116647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.116431    4478 cpu_manager.go:414] "containerMap: removing container" podUID="89fa7173-f13c-4036-93c3-c6f048c6c5f4" containerName="maas-api"
Oct 31 00:14:12.116647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.116448    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fa7173-f13c-4036-93c3-c6f048c6c5f4" containerName="maas-api"
Oct 31 00:14:12.116647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.116467    4478 cpu_manager.go:414] "containerMap: removing container" podUID="afcc8091-a183-435d-be4d-a38aa7507fc0" containerName="authorino"
Oct 31 00:14:12.116647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.116475    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afcc8091-a183-435d-be4d-a38aa7507fc0" containerName="authorino"
Oct 31 00:14:12.116647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.116559    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fa7173-f13c-4036-93c3-c6f048c6c5f4" containerName="maas-api"
Oct 31 00:14:12.116647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.116572    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="afcc8091-a183-435d-be4d-a38aa7507fc0" containerName="authorino"
Oct 31 00:14:12.119416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.119372    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.173022 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.172992    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Oct 31 00:14:12.222494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.222461    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-48j6w\""
Oct 31 00:14:12.222494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.222475    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Oct 31 00:14:12.223881 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.223864    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-txds2"]
Oct 31 00:14:12.241569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.241528    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.241708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.241625    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zmkd\" (UniqueName: \"kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.342849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.342809    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.343019 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.342861    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zmkd\" (UniqueName: \"kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.343019 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:14:12.342950    4478 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found
Oct 31 00:14:12.343019 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:14:12.343012    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs podName:cd97ffc1-8978-4003-aa69-566b5a78ea9c nodeName:}" failed. No retries permitted until 2026-10-31 00:14:12.842994295 +0000 UTC m=+685.859658986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs") pod "ai-gateway-controller-7d85675f98-txds2" (UID: "cd97ffc1-8978-4003-aa69-566b5a78ea9c") : secret "ai-gateway-controller-metrics-tls" not found
Oct 31 00:14:12.404651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.404582    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zmkd\" (UniqueName: \"kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.847261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.847226    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:12.849759 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:12.849728    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\") pod \"ai-gateway-controller-7d85675f98-txds2\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") " pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:13.034299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:13.034272    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:13.402917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:13.402884    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-txds2"]
Oct 31 00:14:13.406746 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:14:13.406720    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd97ffc1_8978_4003_aa69_566b5a78ea9c.slice/crio-a976f43cc5db2ade64de3782c5c52b6c413df857a21bc9e312fb3eb130ef93da WatchSource:0}: Error finding container a976f43cc5db2ade64de3782c5c52b6c413df857a21bc9e312fb3eb130ef93da: Status 404 returned error can't find the container with id a976f43cc5db2ade64de3782c5c52b6c413df857a21bc9e312fb3eb130ef93da
Oct 31 00:14:13.535286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:13.535249    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2" event={"ID":"cd97ffc1-8978-4003-aa69-566b5a78ea9c","Type":"ContainerStarted","Data":"a976f43cc5db2ade64de3782c5c52b6c413df857a21bc9e312fb3eb130ef93da"}
Oct 31 00:14:14.761281 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:14:14.761237    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:14:15.544233 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:15.544200    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2" event={"ID":"cd97ffc1-8978-4003-aa69-566b5a78ea9c","Type":"ContainerStarted","Data":"2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a"}
Oct 31 00:14:15.544431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:15.544281    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:15.664990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:15.664899    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2" podStartSLOduration=1.761411661 podStartE2EDuration="3.664885527s" totalImagesPullingTime="1.903473866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:15.664208382 +0000 UTC m=+688.680873105" watchObservedRunningTime="2026-10-31 00:14:15.664885527 +0000 UTC m=+688.681550238"
Oct 31 00:14:19.638293 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.638258    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"]
Oct 31 00:14:19.641736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.641713    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:19.693824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.693800    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Oct 31 00:14:19.743255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.743230    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-49fm8\""
Oct 31 00:14:19.744539 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.744520    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"]
Oct 31 00:14:19.793871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.793824    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w99g7"]
Oct 31 00:14:19.799474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.798686    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:19.806597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.806570    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13732299-1c22-45fb-8d16-b468502d76f3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qn4q8\" (UID: \"13732299-1c22-45fb-8d16-b468502d76f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:19.806736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.806674    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwjf\" (UniqueName: \"kubernetes.io/projected/13732299-1c22-45fb-8d16-b468502d76f3-kube-api-access-mbwjf\") pod \"payload-pre-processing-7b98bb678c-qn4q8\" (UID: \"13732299-1c22-45fb-8d16-b468502d76f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:19.848858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.848833    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w99g7"]
Oct 31 00:14:19.907269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.907196    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4ltn\" (UniqueName: \"kubernetes.io/projected/c23690f3-509a-470d-8717-5d144235a21a-kube-api-access-l4ltn\") pod \"payload-processing-55fbbbf9dc-w99g7\" (UID: \"c23690f3-509a-470d-8717-5d144235a21a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:19.907269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.907239    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13732299-1c22-45fb-8d16-b468502d76f3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qn4q8\" (UID: \"13732299-1c22-45fb-8d16-b468502d76f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:19.907269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.907259    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwjf\" (UniqueName: \"kubernetes.io/projected/13732299-1c22-45fb-8d16-b468502d76f3-kube-api-access-mbwjf\") pod \"payload-pre-processing-7b98bb678c-qn4q8\" (UID: \"13732299-1c22-45fb-8d16-b468502d76f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:19.907533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.907470    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23690f3-509a-470d-8717-5d144235a21a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w99g7\" (UID: \"c23690f3-509a-470d-8717-5d144235a21a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:19.908007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.907984    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13732299-1c22-45fb-8d16-b468502d76f3-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-qn4q8\" (UID: \"13732299-1c22-45fb-8d16-b468502d76f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:19.968543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:19.968518    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwjf\" (UniqueName: \"kubernetes.io/projected/13732299-1c22-45fb-8d16-b468502d76f3-kube-api-access-mbwjf\") pod \"payload-pre-processing-7b98bb678c-qn4q8\" (UID: \"13732299-1c22-45fb-8d16-b468502d76f3\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:20.008374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.008350    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4ltn\" (UniqueName: \"kubernetes.io/projected/c23690f3-509a-470d-8717-5d144235a21a-kube-api-access-l4ltn\") pod \"payload-processing-55fbbbf9dc-w99g7\" (UID: \"c23690f3-509a-470d-8717-5d144235a21a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:20.008503 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.008445    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23690f3-509a-470d-8717-5d144235a21a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w99g7\" (UID: \"c23690f3-509a-470d-8717-5d144235a21a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:20.009007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.008991    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c23690f3-509a-470d-8717-5d144235a21a-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-w99g7\" (UID: \"c23690f3-509a-470d-8717-5d144235a21a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:20.069163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.069135    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4ltn\" (UniqueName: \"kubernetes.io/projected/c23690f3-509a-470d-8717-5d144235a21a-kube-api-access-l4ltn\") pod \"payload-processing-55fbbbf9dc-w99g7\" (UID: \"c23690f3-509a-470d-8717-5d144235a21a\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:20.113437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.113410    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:20.253312 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.253283    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:20.483013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.482981    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-w99g7"]
Oct 31 00:14:20.486024 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:14:20.485997    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc23690f3_509a_470d_8717_5d144235a21a.slice/crio-56ea442e2aadaa83801ae888ad96400adcfd0b7d77e56b671eee477f5406d1d5 WatchSource:0}: Error finding container 56ea442e2aadaa83801ae888ad96400adcfd0b7d77e56b671eee477f5406d1d5: Status 404 returned error can't find the container with id 56ea442e2aadaa83801ae888ad96400adcfd0b7d77e56b671eee477f5406d1d5
Oct 31 00:14:20.487817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.487804    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:14:20.563879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.563849    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7" event={"ID":"c23690f3-509a-470d-8717-5d144235a21a","Type":"ContainerStarted","Data":"56ea442e2aadaa83801ae888ad96400adcfd0b7d77e56b671eee477f5406d1d5"}
Oct 31 00:14:20.624473 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:20.624450    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"]
Oct 31 00:14:20.627286 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:14:20.627264    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13732299_1c22_45fb_8d16_b468502d76f3.slice/crio-43725c5109d40d4334403684a2d051fc6250bd3e5205e29799c762a2a4011611 WatchSource:0}: Error finding container 43725c5109d40d4334403684a2d051fc6250bd3e5205e29799c762a2a4011611: Status 404 returned error can't find the container with id 43725c5109d40d4334403684a2d051fc6250bd3e5205e29799c762a2a4011611
Oct 31 00:14:21.571350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:21.571315    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8" event={"ID":"13732299-1c22-45fb-8d16-b468502d76f3","Type":"ContainerStarted","Data":"43725c5109d40d4334403684a2d051fc6250bd3e5205e29799c762a2a4011611"}
Oct 31 00:14:23.581720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:23.581624    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7" event={"ID":"c23690f3-509a-470d-8717-5d144235a21a","Type":"ContainerStarted","Data":"57895b482d193a586fff2b409799ad9ee7324da6c53afba95bdc7d77a28a6eae"}
Oct 31 00:14:23.582151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:23.581784    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:23.583186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:23.583162    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8" event={"ID":"13732299-1c22-45fb-8d16-b468502d76f3","Type":"ContainerStarted","Data":"d975a9f1db7add0ef3f72dfc24fdc1ff3e58a703c0a450e70f0a1887741a1d7e"}
Oct 31 00:14:23.583272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:23.583206    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:23.698046 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:23.698003    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7" podStartSLOduration=1.9305539189999998 podStartE2EDuration="4.697988887s" totalImagesPullingTime="2.767434968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:23.697060904 +0000 UTC m=+696.713725621" watchObservedRunningTime="2026-10-31 00:14:23.697988887 +0000 UTC m=+696.714653599"
Oct 31 00:14:23.812275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:23.812231    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8" podStartSLOduration=2.090534602 podStartE2EDuration="4.812216626s" totalImagesPullingTime="2.721682024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:23.811940031 +0000 UTC m=+696.828604744" watchObservedRunningTime="2026-10-31 00:14:23.812216626 +0000 UTC m=+696.828881341"
Oct 31 00:14:26.550466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:26.550433    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:26.909156 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:26.909081    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-654879bb7c-lhq6m"]
Oct 31 00:14:26.912480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:26.912461    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:26.970534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:26.970508    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-654879bb7c-lhq6m"]
Oct 31 00:14:27.064899 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.064861    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63febb67-eed9-4fda-8104-d832eb9a50a5-metrics-certs\") pod \"ai-gateway-controller-654879bb7c-lhq6m\" (UID: \"63febb67-eed9-4fda-8104-d832eb9a50a5\") " pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.065080 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.064908    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxzx6\" (UniqueName: \"kubernetes.io/projected/63febb67-eed9-4fda-8104-d832eb9a50a5-kube-api-access-kxzx6\") pod \"ai-gateway-controller-654879bb7c-lhq6m\" (UID: \"63febb67-eed9-4fda-8104-d832eb9a50a5\") " pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.166358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.166275    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63febb67-eed9-4fda-8104-d832eb9a50a5-metrics-certs\") pod \"ai-gateway-controller-654879bb7c-lhq6m\" (UID: \"63febb67-eed9-4fda-8104-d832eb9a50a5\") " pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.166358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.166314    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxzx6\" (UniqueName: \"kubernetes.io/projected/63febb67-eed9-4fda-8104-d832eb9a50a5-kube-api-access-kxzx6\") pod \"ai-gateway-controller-654879bb7c-lhq6m\" (UID: \"63febb67-eed9-4fda-8104-d832eb9a50a5\") " pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.168792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.168772    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/63febb67-eed9-4fda-8104-d832eb9a50a5-metrics-certs\") pod \"ai-gateway-controller-654879bb7c-lhq6m\" (UID: \"63febb67-eed9-4fda-8104-d832eb9a50a5\") " pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.227419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.227366    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxzx6\" (UniqueName: \"kubernetes.io/projected/63febb67-eed9-4fda-8104-d832eb9a50a5-kube-api-access-kxzx6\") pod \"ai-gateway-controller-654879bb7c-lhq6m\" (UID: \"63febb67-eed9-4fda-8104-d832eb9a50a5\") " pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.524352 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.524320    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:27.894168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:27.894086    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-654879bb7c-lhq6m"]
Oct 31 00:14:27.898159 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:14:27.898134    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63febb67_eed9_4fda_8104_d832eb9a50a5.slice/crio-9e8c28162984793dca3c7fd12f160db0a028fecbd723a189c36cfe0020589ddb WatchSource:0}: Error finding container 9e8c28162984793dca3c7fd12f160db0a028fecbd723a189c36cfe0020589ddb: Status 404 returned error can't find the container with id 9e8c28162984793dca3c7fd12f160db0a028fecbd723a189c36cfe0020589ddb
Oct 31 00:14:28.605280 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:28.605241    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m" event={"ID":"63febb67-eed9-4fda-8104-d832eb9a50a5","Type":"ContainerStarted","Data":"9e8c28162984793dca3c7fd12f160db0a028fecbd723a189c36cfe0020589ddb"}
Oct 31 00:14:29.610591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:29.610559    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m" event={"ID":"63febb67-eed9-4fda-8104-d832eb9a50a5","Type":"ContainerStarted","Data":"44632d1924db6d56b096efea9f4794b934a164185e1d0296d984b132bc51ec5a"}
Oct 31 00:14:29.610974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:29.610674    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:29.727108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:29.727060    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m" podStartSLOduration=2.855700101 podStartE2EDuration="3.727046634s" totalImagesPullingTime="871.346533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:29.725721949 +0000 UTC m=+702.742386662" watchObservedRunningTime="2026-10-31 00:14:29.727046634 +0000 UTC m=+702.743711346"
Oct 31 00:14:34.590694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:34.590665    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-qn4q8"
Oct 31 00:14:34.591138 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:34.590743    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-w99g7"
Oct 31 00:14:40.617356 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:40.617324    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-654879bb7c-lhq6m"
Oct 31 00:14:40.784051 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:40.784023    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-txds2"]
Oct 31 00:14:40.784243 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:40.784202    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2" podUID="cd97ffc1-8978-4003-aa69-566b5a78ea9c" containerName="manager" containerID="cri-o://2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a" gracePeriod=30
Oct 31 00:14:41.023564 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.023540    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:41.166947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.166874    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd\" (UniqueName: \"kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd\") pod \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") "
Oct 31 00:14:41.167104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.166957    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\") pod \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\" (UID: \"cd97ffc1-8978-4003-aa69-566b5a78ea9c\") "
Oct 31 00:14:41.169122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.169094    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd" pod "cd97ffc1-8978-4003-aa69-566b5a78ea9c" (UID: "cd97ffc1-8978-4003-aa69-566b5a78ea9c"). InnerVolumeSpecName "kube-api-access-5zmkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:14:41.169122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.169116    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs" pod "cd97ffc1-8978-4003-aa69-566b5a78ea9c" (UID: "cd97ffc1-8978-4003-aa69-566b5a78ea9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:14:41.268428 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.268404    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd97ffc1-8978-4003-aa69-566b5a78ea9c-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.268428 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.268428    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zmkd\" (UniqueName: \"kubernetes.io/projected/cd97ffc1-8978-4003-aa69-566b5a78ea9c-kube-api-access-5zmkd\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:14:41.664071 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.664038    4478 generic.go:356] "Generic (PLEG): container finished" podID="cd97ffc1-8978-4003-aa69-566b5a78ea9c" containerID="2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a" exitCode=0
Oct 31 00:14:41.664478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.664095    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2" event={"ID":"cd97ffc1-8978-4003-aa69-566b5a78ea9c","Type":"ContainerDied","Data":"2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a"}
Oct 31 00:14:41.664478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.664097    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2"
Oct 31 00:14:41.664478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.664138    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7d85675f98-txds2" event={"ID":"cd97ffc1-8978-4003-aa69-566b5a78ea9c","Type":"ContainerDied","Data":"a976f43cc5db2ade64de3782c5c52b6c413df857a21bc9e312fb3eb130ef93da"}
Oct 31 00:14:41.664478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.664158    4478 scope.go:122] "RemoveContainer" containerID="2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a"
Oct 31 00:14:41.672828 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.672811    4478 scope.go:122] "RemoveContainer" containerID="2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a"
Oct 31 00:14:41.673094 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:14:41.673078    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a\": container with ID starting with 2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a not found: ID does not exist" containerID="2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a"
Oct 31 00:14:41.673135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.673103    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a"} err="failed to get container status \"2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a\": rpc error: code = NotFound desc = could not find container \"2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a\": container with ID starting with 2327fc5c7b73e9bbb2c4c6cdfe7e9a73ab343b544ec7d8caaa3b78eea3f3e94a not found: ID does not exist"
Oct 31 00:14:41.826748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.826721    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-txds2"]
Oct 31 00:14:41.876405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:41.876366    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7d85675f98-txds2"]
Oct 31 00:14:43.115270 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:43.115236    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd97ffc1-8978-4003-aa69-566b5a78ea9c" path="/var/lib/kubelet/pods/cd97ffc1-8978-4003-aa69-566b5a78ea9c/volumes"
Oct 31 00:14:44.756697 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:14:44.756667    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:14:57.821885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.821854    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"]
Oct 31 00:14:57.822265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.822173    4478 cpu_manager.go:414] "containerMap: removing container" podUID="cd97ffc1-8978-4003-aa69-566b5a78ea9c" containerName="manager"
Oct 31 00:14:57.822265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.822185    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd97ffc1-8978-4003-aa69-566b5a78ea9c" containerName="manager"
Oct 31 00:14:57.822265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.822249    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd97ffc1-8978-4003-aa69-566b5a78ea9c" containerName="manager"
Oct 31 00:14:57.828944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.828926    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.895517 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.895488    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d969\" (UniqueName: \"kubernetes.io/projected/a5baef16-a12f-403b-93f6-32fcb692ed4f-kube-api-access-5d969\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.895664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.895549    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5baef16-a12f-403b-93f6-32fcb692ed4f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.895664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.895615    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.895664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.895650    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.895788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.895671    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.895828 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.895797    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.932471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.932449    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Oct 31 00:14:57.932471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.932463    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Oct 31 00:14:57.932683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.932670    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-4mp6x\""
Oct 31 00:14:57.932747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.932683    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Oct 31 00:14:57.933860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.933841    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"]
Oct 31 00:14:57.996884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.996859    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5baef16-a12f-403b-93f6-32fcb692ed4f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997002 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.996888    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997002 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.996911    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997002 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.996939    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997002 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.996989    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.997033    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d969\" (UniqueName: \"kubernetes.io/projected/a5baef16-a12f-403b-93f6-32fcb692ed4f-kube-api-access-5d969\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997333 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.997315    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.997368    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.997423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.997402    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.999280 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.999258    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5baef16-a12f-403b-93f6-32fcb692ed4f-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:57.999458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:57.999442    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5baef16-a12f-403b-93f6-32fcb692ed4f-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:58.058347 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:58.058319    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d969\" (UniqueName: \"kubernetes.io/projected/a5baef16-a12f-403b-93f6-32fcb692ed4f-kube-api-access-5d969\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-6rd7f\" (UID: \"a5baef16-a12f-403b-93f6-32fcb692ed4f\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:58.140814 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:58.140750    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:14:58.513337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:58.513307    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"]
Oct 31 00:14:58.516493 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:14:58.516471    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5baef16_a12f_403b_93f6_32fcb692ed4f.slice/crio-ded1b451b1b546c63cfb4a54abfce89bfa4084803f1c1733f4da101186579c1d WatchSource:0}: Error finding container ded1b451b1b546c63cfb4a54abfce89bfa4084803f1c1733f4da101186579c1d: Status 404 returned error can't find the container with id ded1b451b1b546c63cfb4a54abfce89bfa4084803f1c1733f4da101186579c1d
Oct 31 00:14:58.730233 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:14:58.730201    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f" event={"ID":"a5baef16-a12f-403b-93f6-32fcb692ed4f","Type":"ContainerStarted","Data":"ded1b451b1b546c63cfb4a54abfce89bfa4084803f1c1733f4da101186579c1d"}
Oct 31 00:15:00.750122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:00.750084    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f" event={"ID":"a5baef16-a12f-403b-93f6-32fcb692ed4f","Type":"ContainerStarted","Data":"d58eaa86d016c57ac3c21aa7c6d290bda2d20050de08435ee965ea079b31e8a2"}
Oct 31 00:15:00.867155 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:00.867102    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f" podStartSLOduration=2.027397297 podStartE2EDuration="3.867089861s" totalImagesPullingTime="1.839692564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:00.866353496 +0000 UTC m=+733.883018217" watchObservedRunningTime="2026-10-31 00:15:00.867089861 +0000 UTC m=+733.883754572"
Oct 31 00:15:01.141475 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:01.141408    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:15:05.181966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.181937    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"]
Oct 31 00:15:05.185094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.185078    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.237470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.237429    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Oct 31 00:15:05.260937 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.260905    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.261132 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.260951    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk6np\" (UniqueName: \"kubernetes.io/projected/e2acb604-817d-4688-a5ce-0c8b755ff607-kube-api-access-dk6np\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.261132 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.261091    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2acb604-817d-4688-a5ce-0c8b755ff607-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.261251 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.261166    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.261251 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.261203    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.261360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.261303    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.288614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.288581    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"]
Oct 31 00:15:05.361961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.361927    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.361983    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362141    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362207    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk6np\" (UniqueName: \"kubernetes.io/projected/e2acb604-817d-4688-a5ce-0c8b755ff607-kube-api-access-dk6np\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362236    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2acb604-817d-4688-a5ce-0c8b755ff607-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362287    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362295    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362375    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.362650 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.362633    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.364357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.364339    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e2acb604-817d-4688-a5ce-0c8b755ff607-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.364842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.364822    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2acb604-817d-4688-a5ce-0c8b755ff607-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.423128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.423102    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk6np\" (UniqueName: \"kubernetes.io/projected/e2acb604-817d-4688-a5ce-0c8b755ff607-kube-api-access-dk6np\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-d2vp6\" (UID: \"e2acb604-817d-4688-a5ce-0c8b755ff607\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.497652 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.497561    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:05.874067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:05.873883    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"]
Oct 31 00:15:05.876706 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:15:05.876669    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2acb604_817d_4688_a5ce_0c8b755ff607.slice/crio-3c0942ed4b127cdf5ebe007de6d99bcb63d7d517cde8b543247c9c4fa3b123e3 WatchSource:0}: Error finding container 3c0942ed4b127cdf5ebe007de6d99bcb63d7d517cde8b543247c9c4fa3b123e3: Status 404 returned error can't find the container with id 3c0942ed4b127cdf5ebe007de6d99bcb63d7d517cde8b543247c9c4fa3b123e3
Oct 31 00:15:06.775953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:06.775871    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6" event={"ID":"e2acb604-817d-4688-a5ce-0c8b755ff607","Type":"ContainerStarted","Data":"d0b5dc631fb9c5dbdbbab2758b85d841539481b7dd4a2ace83ae7efc696a93d9"}
Oct 31 00:15:06.775953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:06.775921    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6" event={"ID":"e2acb604-817d-4688-a5ce-0c8b755ff607","Type":"ContainerStarted","Data":"3c0942ed4b127cdf5ebe007de6d99bcb63d7d517cde8b543247c9c4fa3b123e3"}
Oct 31 00:15:06.895180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:06.895125    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6" podStartSLOduration=1.274989986 podStartE2EDuration="1.895111073s" totalImagesPullingTime="620.121087ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:06.894352718 +0000 UTC m=+739.911017454" watchObservedRunningTime="2026-10-31 00:15:06.895111073 +0000 UTC m=+739.911775836"
Oct 31 00:15:07.498257 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:07.498219    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:08.141840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:08.141807    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:15:08.154272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:08.154251    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:15:08.796793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:08.796765    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-6rd7f"
Oct 31 00:15:14.760973 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:15:14.760929    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:15:15.497962 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:15.497930    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:15.510728 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:15.510703    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:15.823804 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:15.823726    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-d2vp6"
Oct 31 00:15:16.122902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.122832    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qf644"]
Oct 31 00:15:16.126199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.126179    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.178234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.178208    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Oct 31 00:15:16.184970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.184941    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qf644"]
Oct 31 00:15:16.248939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.248895    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-home\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.249101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.248980    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.249101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.249014    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.249101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.249069    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htc95\" (UniqueName: \"kubernetes.io/projected/c87839e0-6c15-4f16-9683-8242fe26d76b-kube-api-access-htc95\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.249250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.249178    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.249250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.249208    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87839e0-6c15-4f16-9683-8242fe26d76b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350346 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350311    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350346 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350344    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htc95\" (UniqueName: \"kubernetes.io/projected/c87839e0-6c15-4f16-9683-8242fe26d76b-kube-api-access-htc95\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350399    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350431    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87839e0-6c15-4f16-9683-8242fe26d76b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350476    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-home\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350499    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350843    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350866    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.350965 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.350899    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-home\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.352635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.352612    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c87839e0-6c15-4f16-9683-8242fe26d76b-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.353117 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.353096    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c87839e0-6c15-4f16-9683-8242fe26d76b-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.415565 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.415495    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htc95\" (UniqueName: \"kubernetes.io/projected/c87839e0-6c15-4f16-9683-8242fe26d76b-kube-api-access-htc95\") pod \"echo-simulator-kserve-64b6f8f96c-qf644\" (UID: \"c87839e0-6c15-4f16-9683-8242fe26d76b\") " pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.438353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.438323    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:16.812452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:16.812422    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-qf644"]
Oct 31 00:15:16.815728 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:15:16.815698    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc87839e0_6c15_4f16_9683_8242fe26d76b.slice/crio-9011bf894c79c2ac0771ef802990c0c22b90269b0b1e320cea06876b1f047265 WatchSource:0}: Error finding container 9011bf894c79c2ac0771ef802990c0c22b90269b0b1e320cea06876b1f047265: Status 404 returned error can't find the container with id 9011bf894c79c2ac0771ef802990c0c22b90269b0b1e320cea06876b1f047265
Oct 31 00:15:17.822418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:17.822361    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644" event={"ID":"c87839e0-6c15-4f16-9683-8242fe26d76b","Type":"ContainerStarted","Data":"9011bf894c79c2ac0771ef802990c0c22b90269b0b1e320cea06876b1f047265"}
Oct 31 00:15:19.752414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.752365    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"]
Oct 31 00:15:19.755922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.755904    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.835682 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.835647    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644" event={"ID":"c87839e0-6c15-4f16-9683-8242fe26d76b","Type":"ContainerStarted","Data":"6e9dbfafbb4b57f6e2d59fc53c5ea96049817a13a00e8943cd61b593bcef4de4"}
Oct 31 00:15:19.857300 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.857277    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Oct 31 00:15:19.858821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.858801    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"]
Oct 31 00:15:19.879388 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.879360    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psb8t\" (UniqueName: \"kubernetes.io/projected/88aa52c9-9c13-4b54-b62d-929ff91b26a7-kube-api-access-psb8t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.879610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.879589    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.879683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.879624    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.879792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.879768    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.879895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.879821    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88aa52c9-9c13-4b54-b62d-929ff91b26a7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.879895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.879855    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980464    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980518    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psb8t\" (UniqueName: \"kubernetes.io/projected/88aa52c9-9c13-4b54-b62d-929ff91b26a7-kube-api-access-psb8t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980541    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980558    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980579    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980596    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88aa52c9-9c13-4b54-b62d-929ff91b26a7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.980924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980900    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.981003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980928    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.981053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.980996    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.982991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.982968    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88aa52c9-9c13-4b54-b62d-929ff91b26a7-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.983114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.983098    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88aa52c9-9c13-4b54-b62d-929ff91b26a7-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:19.988376 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:19.988332    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644" podStartSLOduration=1.636652293 podStartE2EDuration="3.98831947s" totalImagesPullingTime="2.351667177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:19.987503789 +0000 UTC m=+753.004168498" watchObservedRunningTime="2026-10-31 00:15:19.98831947 +0000 UTC m=+753.004984182"
Oct 31 00:15:20.044233 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:20.044174    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psb8t\" (UniqueName: \"kubernetes.io/projected/88aa52c9-9c13-4b54-b62d-929ff91b26a7-kube-api-access-psb8t\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv\" (UID: \"88aa52c9-9c13-4b54-b62d-929ff91b26a7\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:20.068508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:20.068485    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:20.439278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:20.439248    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:20.441456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:20.441430    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"]
Oct 31 00:15:20.444585 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:15:20.444557    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88aa52c9_9c13_4b54_b62d_929ff91b26a7.slice/crio-ddd66325531778031fd3bb0d9ce5422174414ec9fbcba2e2f7651c1d1f29bf3d WatchSource:0}: Error finding container ddd66325531778031fd3bb0d9ce5422174414ec9fbcba2e2f7651c1d1f29bf3d: Status 404 returned error can't find the container with id ddd66325531778031fd3bb0d9ce5422174414ec9fbcba2e2f7651c1d1f29bf3d
Oct 31 00:15:20.840991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:20.840920    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv" event={"ID":"88aa52c9-9c13-4b54-b62d-929ff91b26a7","Type":"ContainerStarted","Data":"ddd66325531778031fd3bb0d9ce5422174414ec9fbcba2e2f7651c1d1f29bf3d"}
Oct 31 00:15:21.846951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:21.846917    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv" event={"ID":"88aa52c9-9c13-4b54-b62d-929ff91b26a7","Type":"ContainerStarted","Data":"a9ce242e58ce8c330ca9782b87b40f377f827b25b4b9159ec78d875145b3ae80"}
Oct 31 00:15:21.964611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:21.964557    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv" podStartSLOduration=2.399385871 podStartE2EDuration="2.964542557s" totalImagesPullingTime="565.156686ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:21.962940645 +0000 UTC m=+754.979605375" watchObservedRunningTime="2026-10-31 00:15:21.964542557 +0000 UTC m=+754.981207268"
Oct 31 00:15:22.069056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:22.069023    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:26.438521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:26.438486    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:26.450556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:26.450536    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:26.878204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:26.878126    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-qf644"
Oct 31 00:15:30.069071 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:30.068985    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:30.081413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:30.081368    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:30.894237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:30.894209    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv"
Oct 31 00:15:41.492210 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.492176    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"]
Oct 31 00:15:41.494435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.494419    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.546591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.546563    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-r4dfh\""
Oct 31 00:15:41.658514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658477    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkz7\" (UniqueName: \"kubernetes.io/projected/fa40b2b4-590d-4f48-8363-58137eb2e5bc-kube-api-access-blkz7\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658513    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658531    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658549    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658701    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658741    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658834    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.658921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658910    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.659053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.658956    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.697889 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.697560    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"]
Oct 31 00:15:41.759608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759542    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759585    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759605    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759625    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759654    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759683    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759744    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkz7\" (UniqueName: \"kubernetes.io/projected/fa40b2b4-590d-4f48-8363-58137eb2e5bc-kube-api-access-blkz7\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759771    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.759857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.759792    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.760140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.760056    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.760140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.760077    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.760140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.760119    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.760253 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.760230    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.760560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.760538    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.762077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.762052    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.762638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.762620    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.820920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.820893    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa40b2b4-590d-4f48-8363-58137eb2e5bc-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:41.821332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:41.821310    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkz7\" (UniqueName: \"kubernetes.io/projected/fa40b2b4-590d-4f48-8363-58137eb2e5bc-kube-api-access-blkz7\") pod \"maas-default-gateway-openshift-default-58c6585447-7ljch\" (UID: \"fa40b2b4-590d-4f48-8363-58137eb2e5bc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:42.105607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:42.105524    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:42.534701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:42.534678    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"]
Oct 31 00:15:42.537224 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:15:42.537180    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa40b2b4_590d_4f48_8363_58137eb2e5bc.slice/crio-01dcc723ceec60f888ae381fc73df1f7013f4be96780ec0575738497ab23f7d9 WatchSource:0}: Error finding container 01dcc723ceec60f888ae381fc73df1f7013f4be96780ec0575738497ab23f7d9: Status 404 returned error can't find the container with id 01dcc723ceec60f888ae381fc73df1f7013f4be96780ec0575738497ab23f7d9
Oct 31 00:15:42.935540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:42.935509    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" event={"ID":"fa40b2b4-590d-4f48-8363-58137eb2e5bc","Type":"ContainerStarted","Data":"01dcc723ceec60f888ae381fc73df1f7013f4be96780ec0575738497ab23f7d9"}
Oct 31 00:15:44.759854 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:15:44.759819    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:15:46.384714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:46.384681    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:15:46.385045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:46.384748    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:15:46.385045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:46.384776    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:15:46.960099 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:46.960063    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" event={"ID":"fa40b2b4-590d-4f48-8363-58137eb2e5bc","Type":"ContainerStarted","Data":"ab65a01deac18aaea69b3925e3e3ca6b374bcafb4d03c2bb57d25b3646d46993"}
Oct 31 00:15:47.106353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:47.106317    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:47.108087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:47.108063    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:47.108184 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:47.108116    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:47.130630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:47.130586    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podStartSLOduration=2.286011988 podStartE2EDuration="6.130571649s" totalImagesPullingTime="3.844559661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:47.13019404 +0000 UTC m=+780.146858775" watchObservedRunningTime="2026-10-31 00:15:47.130571649 +0000 UTC m=+780.147236361"
Oct 31 00:15:48.105869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:48.105838    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:48.106277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:48.105890    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:49.106655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.106619    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:49.107045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.106677    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:49.456541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.456501    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5f4bb8f767-9xcl4"]
Oct 31 00:15:49.460353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.460333    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:49.510714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.510688    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4bb8f767-9xcl4"]
Oct 31 00:15:49.558604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.558552    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-7p7cn\""
Oct 31 00:15:49.561881 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.561860    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 31 00:15:49.630587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.630545    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/450dc4e8-04b1-40cb-89a7-71d635a80d9d-tls-cert\") pod \"authorino-5f4bb8f767-9xcl4\" (UID: \"450dc4e8-04b1-40cb-89a7-71d635a80d9d\") " pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:49.630721 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.630611    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnwzz\" (UniqueName: \"kubernetes.io/projected/450dc4e8-04b1-40cb-89a7-71d635a80d9d-kube-api-access-tnwzz\") pod \"authorino-5f4bb8f767-9xcl4\" (UID: \"450dc4e8-04b1-40cb-89a7-71d635a80d9d\") " pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:49.731536 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.731456    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/450dc4e8-04b1-40cb-89a7-71d635a80d9d-tls-cert\") pod \"authorino-5f4bb8f767-9xcl4\" (UID: \"450dc4e8-04b1-40cb-89a7-71d635a80d9d\") " pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:49.732238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.732215    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnwzz\" (UniqueName: \"kubernetes.io/projected/450dc4e8-04b1-40cb-89a7-71d635a80d9d-kube-api-access-tnwzz\") pod \"authorino-5f4bb8f767-9xcl4\" (UID: \"450dc4e8-04b1-40cb-89a7-71d635a80d9d\") " pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:49.746762 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.746739    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/450dc4e8-04b1-40cb-89a7-71d635a80d9d-tls-cert\") pod \"authorino-5f4bb8f767-9xcl4\" (UID: \"450dc4e8-04b1-40cb-89a7-71d635a80d9d\") " pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:49.793992 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:49.793965    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnwzz\" (UniqueName: \"kubernetes.io/projected/450dc4e8-04b1-40cb-89a7-71d635a80d9d-kube-api-access-tnwzz\") pod \"authorino-5f4bb8f767-9xcl4\" (UID: \"450dc4e8-04b1-40cb-89a7-71d635a80d9d\") " pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:50.080073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:50.079999    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5f4bb8f767-9xcl4"
Oct 31 00:15:50.106510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:50.106480    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:50.106660 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:50.106539    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:50.481355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:50.481163    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5f4bb8f767-9xcl4"]
Oct 31 00:15:50.484152 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:15:50.484124    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod450dc4e8_04b1_40cb_89a7_71d635a80d9d.slice/crio-f880224636464a22a193a0639eddb0bd7a755305dcfb445c21f451ac431ccef9 WatchSource:0}: Error finding container f880224636464a22a193a0639eddb0bd7a755305dcfb445c21f451ac431ccef9: Status 404 returned error can't find the container with id f880224636464a22a193a0639eddb0bd7a755305dcfb445c21f451ac431ccef9
Oct 31 00:15:50.978938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:50.978894    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4bb8f767-9xcl4" event={"ID":"450dc4e8-04b1-40cb-89a7-71d635a80d9d","Type":"ContainerStarted","Data":"f880224636464a22a193a0639eddb0bd7a755305dcfb445c21f451ac431ccef9"}
Oct 31 00:15:51.106685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:51.106650    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:51.106848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:51.106711    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:52.107906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:52.105870    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:52.107906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:52.105916    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:52.989945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:52.989907    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5f4bb8f767-9xcl4" event={"ID":"450dc4e8-04b1-40cb-89a7-71d635a80d9d","Type":"ContainerStarted","Data":"f9035d4ad6701c4a5494794aa5d42ab799c3c74a60252fdf607e7703ed81ba05"}
Oct 31 00:15:53.105611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:53.105544    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5f4bb8f767-9xcl4" podStartSLOduration=2.496400388 podStartE2EDuration="4.105522594s" totalImagesPullingTime="1.609122206s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:53.104302306 +0000 UTC m=+786.120967050" watchObservedRunningTime="2026-10-31 00:15:53.105522594 +0000 UTC m=+786.122187307"
Oct 31 00:15:53.106783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:53.106752    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-7ljch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Oct 31 00:15:53.106924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:53.106822    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch" podUID="fa40b2b4-590d-4f48-8363-58137eb2e5bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Oct 31 00:15:54.735572 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:54.735537    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:54.736563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:54.736536    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:54.737510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:54.737478    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-7ljch"
Oct 31 00:15:56.592739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.592703    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"]
Oct 31 00:15:56.596215 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.596199    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.613662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613635    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.613792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613675    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.613792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613777    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.613898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613829    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.613933 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613905    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.613988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613968    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7tdr\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.614023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.613999    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.614023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.614015    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.614149 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.614040    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.699770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.699732    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"]
Oct 31 00:15:56.714952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.714913    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7tdr\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.714963    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.714991    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715029    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715171    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715218    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715368 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715270    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715368 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715329    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715368 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715364    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715487    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715754    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715778    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715754    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.715866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.715814    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.718096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.718075    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.718590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.718568    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.775805 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.775773    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.775943 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.775831    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7tdr\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr\") pod \"maas-default-gateway-openshift-default-58c6585447-4kf6r\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:56.909534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:56.909462    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:57.336583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:57.336551    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"]
Oct 31 00:15:57.340096 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:15:57.340073    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf5bcbc_4566_490d_a288_f1661106fb7b.slice/crio-c4dcb4cd7f1ed112bff0a86f2dcc5afb73ff118320e1af2143f787ba10042043 WatchSource:0}: Error finding container c4dcb4cd7f1ed112bff0a86f2dcc5afb73ff118320e1af2143f787ba10042043: Status 404 returned error can't find the container with id c4dcb4cd7f1ed112bff0a86f2dcc5afb73ff118320e1af2143f787ba10042043
Oct 31 00:15:57.342554 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:57.342513    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:15:57.342628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:57.342605    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:15:57.342665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:57.342649    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:15:58.025199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:58.025162    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" event={"ID":"0cf5bcbc-4566-490d-a288-f1661106fb7b","Type":"ContainerStarted","Data":"3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82"}
Oct 31 00:15:58.025623 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:58.025241    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" event={"ID":"0cf5bcbc-4566-490d-a288-f1661106fb7b","Type":"ContainerStarted","Data":"c4dcb4cd7f1ed112bff0a86f2dcc5afb73ff118320e1af2143f787ba10042043"}
Oct 31 00:15:58.195345 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:58.195290    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podStartSLOduration=2.195275751 podStartE2EDuration="2.195275751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:58.193842097 +0000 UTC m=+791.210506813" watchObservedRunningTime="2026-10-31 00:15:58.195275751 +0000 UTC m=+791.211940463"
Oct 31 00:15:58.910056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:58.910015    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:15:58.911626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:58.911600    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4kf6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 00:15:58.911755 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:58.911660    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 00:15:59.910498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:59.910465    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4kf6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 00:15:59.910884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:15:59.910527    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 00:16:00.910006 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:00.909967    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4kf6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 00:16:00.910177 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:00.910030    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 00:16:01.910430 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:01.910363    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4kf6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 00:16:01.910853 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:01.910462    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 00:16:02.910017 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:02.909983    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4kf6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 00:16:02.910204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:02.910032    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 00:16:03.910400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:03.910350    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4kf6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused" start-of-body=
Oct 31 00:16:03.910885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:03.910427    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.38:15021/healthz/ready\": dial tcp 10.132.0.38:15021: connect: connection refused"
Oct 31 00:16:05.572176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:05.572129    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:16:05.572674 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:05.572431    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:16:05.573453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:05.573431    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:16:14.755503 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:14.755464    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:16:28.810917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.810885    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"]
Oct 31 00:16:28.817803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.817783    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.870062 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.870032    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-6ab60e97-openshift-default-dockercfg-hnps4\""
Oct 31 00:16:28.915479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915448    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915488    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915522    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915709 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915631    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915709 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915669    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msbbs\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915774 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915757    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915820 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915806    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915849    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.915905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.915874    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:28.921056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:28.921038    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"]
Oct 31 00:16:29.016500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016477    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016511    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msbbs\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016539    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016659    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016705    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016734    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016929 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016818    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016929 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016845    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.016929 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.016882    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.017075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.017032    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.017130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.017068    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.017213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.017192    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.017433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.017414    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.017488 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.017453    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.018890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.018866    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.019364 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.019347    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.077550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.077484    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.077677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.077571    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msbbs\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs\") pod \"e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") " pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.130949 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.130925    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:29.555916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.555796    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"]
Oct 31 00:16:29.558617 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:29.558591    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc503cfa5_d727_475b_a32c_df9626426349.slice/crio-b58c832722add47851a88e2ed87e9b7bdbb960837576f0c618ec82682690e6d8 WatchSource:0}: Error finding container b58c832722add47851a88e2ed87e9b7bdbb960837576f0c618ec82682690e6d8: Status 404 returned error can't find the container with id b58c832722add47851a88e2ed87e9b7bdbb960837576f0c618ec82682690e6d8
Oct 31 00:16:29.560684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.560654    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:16:29.560758 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.560715    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:16:29.560758 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:29.560744    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:16:30.177614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:30.177577    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" event={"ID":"c503cfa5-d727-475b-a32c-df9626426349","Type":"ContainerStarted","Data":"18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"}
Oct 31 00:16:30.177990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:30.177668    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" event={"ID":"c503cfa5-d727-475b-a32c-df9626426349","Type":"ContainerStarted","Data":"b58c832722add47851a88e2ed87e9b7bdbb960837576f0c618ec82682690e6d8"}
Oct 31 00:16:30.303628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:30.303571    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podStartSLOduration=2.303559725 podStartE2EDuration="2.303559725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:30.303020866 +0000 UTC m=+823.319685578" watchObservedRunningTime="2026-10-31 00:16:30.303559725 +0000 UTC m=+823.320224436"
Oct 31 00:16:31.131558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:31.131518    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:31.137140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:31.137113    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:31.182202 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:31.182170    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:31.183055 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:31.183035    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:16:34.036049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.036014    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"]
Oct 31 00:16:34.039536 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.039516    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.092848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.092821    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4b78c516\""
Oct 31 00:16:34.144405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.144356    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"]
Oct 31 00:16:34.166216 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.166189    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.166429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.166237    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwchg\" (UniqueName: \"kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg\") pod \"payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.187971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.187941    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"]
Oct 31 00:16:34.191271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.191255    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.191337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.191294    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-4b78c516-dockercfg-5kccv\""
Oct 31 00:16:34.229772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.229744    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"]
Oct 31 00:16:34.267484 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.267455    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhvr\" (UniqueName: \"kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr\") pod \"payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.267710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.267659    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.267875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.267764    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.267875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.267828    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwchg\" (UniqueName: \"kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg\") pod \"payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.268665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.268644    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.331399 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.331327    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwchg\" (UniqueName: \"kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg\") pod \"payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.352176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.352154    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:34.368587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.368545    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhvr\" (UniqueName: \"kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr\") pod \"payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.368677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.368654    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.369353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.369335    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.429791 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.429758    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhvr\" (UniqueName: \"kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr\") pod \"payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") " pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.503831 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.503802    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:34.726663 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.726632    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"]
Oct 31 00:16:34.729680 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:34.729654    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c87b23_99d2_4350_9743_7f60f739dfa8.slice/crio-6909d0015c6b1dbbc6f664155987138a7901ab0c47e0867e82d67158e5ed692c WatchSource:0}: Error finding container 6909d0015c6b1dbbc6f664155987138a7901ab0c47e0867e82d67158e5ed692c: Status 404 returned error can't find the container with id 6909d0015c6b1dbbc6f664155987138a7901ab0c47e0867e82d67158e5ed692c
Oct 31 00:16:34.897206 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:34.897183    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"]
Oct 31 00:16:34.899730 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:34.899700    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc794ec8b_c7d0_4e0c_a7be_deaf365920ec.slice/crio-d4be8f59d1cd841875cd560842f914f3d7a4c9d69a574a5fa7ef91de5b88d8c3 WatchSource:0}: Error finding container d4be8f59d1cd841875cd560842f914f3d7a4c9d69a574a5fa7ef91de5b88d8c3: Status 404 returned error can't find the container with id d4be8f59d1cd841875cd560842f914f3d7a4c9d69a574a5fa7ef91de5b88d8c3
Oct 31 00:16:35.200218 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.200181    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q" event={"ID":"c794ec8b-c7d0-4e0c-a7be-deaf365920ec","Type":"ContainerStarted","Data":"847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452"}
Oct 31 00:16:35.200691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.200260    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q" event={"ID":"c794ec8b-c7d0-4e0c-a7be-deaf365920ec","Type":"ContainerStarted","Data":"d4be8f59d1cd841875cd560842f914f3d7a4c9d69a574a5fa7ef91de5b88d8c3"}
Oct 31 00:16:35.200691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.200417    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:35.201750 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.201730    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t" event={"ID":"46c87b23-99d2-4350-9743-7f60f739dfa8","Type":"ContainerStarted","Data":"e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d"}
Oct 31 00:16:35.201860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.201763    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t" event={"ID":"46c87b23-99d2-4350-9743-7f60f739dfa8","Type":"ContainerStarted","Data":"6909d0015c6b1dbbc6f664155987138a7901ab0c47e0867e82d67158e5ed692c"}
Oct 31 00:16:35.201860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.201782    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:35.315704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.315663    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q" podStartSLOduration=1.315652435 podStartE2EDuration="1.315652435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:35.315005303 +0000 UTC m=+828.331670063" watchObservedRunningTime="2026-10-31 00:16:35.315652435 +0000 UTC m=+828.332317146"
Oct 31 00:16:35.446549 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.446491    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t" podStartSLOduration=2.446475462 podStartE2EDuration="2.446475462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:35.445066289 +0000 UTC m=+828.461731003" watchObservedRunningTime="2026-10-31 00:16:35.446475462 +0000 UTC m=+828.463140174"
Oct 31 00:16:35.904546 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.904508    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"]
Oct 31 00:16:35.908133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.908110    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:35.960267 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.960239    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e1bbe4b-serving-cert\""
Oct 31 00:16:35.967039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.967018    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"]
Oct 31 00:16:35.983186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.983163    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cg8zm\" (UniqueName: \"kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:35.983306 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.983199    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:35.983360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.983316    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:35.983431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:35.983369    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.084052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.084021    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.084234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.084061    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cg8zm\" (UniqueName: \"kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.084234 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:36.084178    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-serving-cert: secret "maas-api-e2e-ait-0e1bbe4b-serving-cert" not found
Oct 31 00:16:36.084367 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:36.084261    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls podName:a94ef019-4b6b-4f1b-afe9-18ea5120fb42 nodeName:}" failed. No retries permitted until 2026-10-31 00:16:36.584239517 +0000 UTC m=+829.600904216 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls") pod "maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" (UID: "a94ef019-4b6b-4f1b-afe9-18ea5120fb42") : secret "maas-api-e2e-ait-0e1bbe4b-serving-cert" not found
Oct 31 00:16:36.084367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.084179    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.084533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.084421    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.085097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.085040    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.086784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.086764    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.145139 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.145107    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cg8zm\" (UniqueName: \"kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.588865 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.588830    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.592447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.592420    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\") pod \"maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:36.821142 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:36.821110    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:37.226498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:37.226476    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"]
Oct 31 00:16:37.228637 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:37.228600    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda94ef019_4b6b_4f1b_afe9_18ea5120fb42.slice/crio-847a6c9581617d58c8c090a27f031a78a2b3337f33ed088379f3d02f453488b7 WatchSource:0}: Error finding container 847a6c9581617d58c8c090a27f031a78a2b3337f33ed088379f3d02f453488b7: Status 404 returned error can't find the container with id 847a6c9581617d58c8c090a27f031a78a2b3337f33ed088379f3d02f453488b7
Oct 31 00:16:38.219192 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:38.219153    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" event={"ID":"a94ef019-4b6b-4f1b-afe9-18ea5120fb42","Type":"ContainerStarted","Data":"847a6c9581617d58c8c090a27f031a78a2b3337f33ed088379f3d02f453488b7"}
Oct 31 00:16:39.225847 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:39.225798    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" event={"ID":"a94ef019-4b6b-4f1b-afe9-18ea5120fb42","Type":"ContainerStarted","Data":"c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d"}
Oct 31 00:16:39.226252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:39.225873    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:39.343095 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:39.343049    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" podStartSLOduration=3.391888465 podStartE2EDuration="4.343034359s" totalImagesPullingTime="951.145894ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:39.3421745 +0000 UTC m=+832.358839212" watchObservedRunningTime="2026-10-31 00:16:39.343034359 +0000 UTC m=+832.359699070"
Oct 31 00:16:41.402606 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.402576    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"]
Oct 31 00:16:41.406412 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.406363    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.458612 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.458577    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4b78c516-serving-cert\""
Oct 31 00:16:41.510798 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.510763    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"]
Oct 31 00:16:41.540563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.540534    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.540696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.540588    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgvc9\" (UniqueName: \"kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.540696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.540673    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.540799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.540733    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.641878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.641838    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgvc9\" (UniqueName: \"kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.643064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.641900    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.643064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.641962    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.643064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.642042    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.643064 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:41.642171    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-serving-cert: secret "maas-api-e2e-migrate-4b78c516-serving-cert" not found
Oct 31 00:16:41.643064 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:41.642236    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls podName:b74374be-0eed-4bba-8059-bd563271eeef nodeName:}" failed. No retries permitted until 2026-10-31 00:16:42.14221609 +0000 UTC m=+835.158880800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls") pod "maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" (UID: "b74374be-0eed-4bba-8059-bd563271eeef") : secret "maas-api-e2e-migrate-4b78c516-serving-cert" not found
Oct 31 00:16:41.644326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.643359    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.646712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.646299    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:41.706491 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:41.704546    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgvc9\" (UniqueName: \"kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:42.148499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:42.148420    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:42.151064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:42.151036    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\") pod \"maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:42.325880 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:42.325840    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:42.699350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:42.699281    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"]
Oct 31 00:16:42.701899 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:42.701870    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb74374be_0eed_4bba_8059_bd563271eeef.slice/crio-e6e5d6dfd3548704f28bc2fa9ddaaf106c1481557486d3abff3bf472a4bf45ac WatchSource:0}: Error finding container e6e5d6dfd3548704f28bc2fa9ddaaf106c1481557486d3abff3bf472a4bf45ac: Status 404 returned error can't find the container with id e6e5d6dfd3548704f28bc2fa9ddaaf106c1481557486d3abff3bf472a4bf45ac
Oct 31 00:16:43.245358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:43.245326    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" event={"ID":"b74374be-0eed-4bba-8059-bd563271eeef","Type":"ContainerStarted","Data":"e6e5d6dfd3548704f28bc2fa9ddaaf106c1481557486d3abff3bf472a4bf45ac"}
Oct 31 00:16:44.104738 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.104699    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"]
Oct 31 00:16:44.108256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.108235    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.210630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.210594    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6ab60e97\""
Oct 31 00:16:44.210838 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.210781    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6ab60e97-dockercfg-854wx\""
Oct 31 00:16:44.218403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.214330    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"]
Oct 31 00:16:44.252736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.252703    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" event={"ID":"b74374be-0eed-4bba-8059-bd563271eeef","Type":"ContainerStarted","Data":"38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117"}
Oct 31 00:16:44.252906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.252833    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:44.267005 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.266979    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.267121 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.267024    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgd2p\" (UniqueName: \"kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p\") pod \"payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.368000 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.367925    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.368150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.368015    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgd2p\" (UniqueName: \"kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p\") pod \"payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.368677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.368656    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.371361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.371305    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" podStartSLOduration=2.5230617889999998 podStartE2EDuration="3.371292507s" totalImagesPullingTime="848.230718ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:44.370587862 +0000 UTC m=+837.387252575" watchObservedRunningTime="2026-10-31 00:16:44.371292507 +0000 UTC m=+837.387957219"
Oct 31 00:16:44.429767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.429743    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgd2p\" (UniqueName: \"kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p\") pod \"payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.721011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:44.720983    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:44.759804 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:44.759764    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:16:45.102094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:45.102014    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"]
Oct 31 00:16:45.104675 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:45.104646    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d778103_d6a8_4f52_8848_c28d4c6c4955.slice/crio-2ad5949569bbcd2af9d7ccedaa680256089e1e3134ab0dbab65ac293ffeef05f WatchSource:0}: Error finding container 2ad5949569bbcd2af9d7ccedaa680256089e1e3134ab0dbab65ac293ffeef05f: Status 404 returned error can't find the container with id 2ad5949569bbcd2af9d7ccedaa680256089e1e3134ab0dbab65ac293ffeef05f
Oct 31 00:16:45.237303 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:45.237276    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:16:45.259153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:45.259124    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh" event={"ID":"9d778103-d6a8-4f52-8848-c28d4c6c4955","Type":"ContainerStarted","Data":"124bf7a104c55c634e52723c69165b88ffd6c237760f1c6fc4adad8a6e62fdcf"}
Oct 31 00:16:45.259286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:45.259179    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh" event={"ID":"9d778103-d6a8-4f52-8848-c28d4c6c4955","Type":"ContainerStarted","Data":"2ad5949569bbcd2af9d7ccedaa680256089e1e3134ab0dbab65ac293ffeef05f"}
Oct 31 00:16:45.259286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:45.259210    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:45.512019 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:45.511970    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh" podStartSLOduration=1.5119564159999999 podStartE2EDuration="1.511956416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:45.509937122 +0000 UTC m=+838.526601851" watchObservedRunningTime="2026-10-31 00:16:45.511956416 +0000 UTC m=+838.528621170"
Oct 31 00:16:46.206248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.206216    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"]
Oct 31 00:16:46.210007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.209984    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:16:46.210141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.210027    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:16:46.210215 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.210171    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.262174 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.262148    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6ab60e97-serving-cert\""
Oct 31 00:16:46.313293 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.313264    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"]
Oct 31 00:16:46.386068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.386039    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.386457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.386427    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.386599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.386499    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2f8z\" (UniqueName: \"kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.386599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.386571    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.487871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.487787    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.487871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.487842    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2f8z\" (UniqueName: \"kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.488068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.487893    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.488068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.487936    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.488548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.488521    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.490599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.490573    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.490705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.490687    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.548626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.548589    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2f8z\" (UniqueName: \"kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z\") pod \"maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:46.822466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:46.822372    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:47.202586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:47.202478    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"]
Oct 31 00:16:47.204972 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:47.204946    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942ec4be_d10a_4c44_ae69_2495548e4330.slice/crio-5e9dc56ecea535f94eb65cd1e1cc76f03f0256eb87cdbc8390850d157a64152f WatchSource:0}: Error finding container 5e9dc56ecea535f94eb65cd1e1cc76f03f0256eb87cdbc8390850d157a64152f: Status 404 returned error can't find the container with id 5e9dc56ecea535f94eb65cd1e1cc76f03f0256eb87cdbc8390850d157a64152f
Oct 31 00:16:47.268892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:47.268861    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d" event={"ID":"942ec4be-d10a-4c44-ae69-2495548e4330","Type":"ContainerStarted","Data":"5e9dc56ecea535f94eb65cd1e1cc76f03f0256eb87cdbc8390850d157a64152f"}
Oct 31 00:16:48.274840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:48.274741    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d" event={"ID":"942ec4be-d10a-4c44-ae69-2495548e4330","Type":"ContainerStarted","Data":"4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93"}
Oct 31 00:16:48.275283 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:48.274874    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:48.392186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:48.392131    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d" podStartSLOduration=1.5983006 podStartE2EDuration="2.392116233s" totalImagesPullingTime="793.815633ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:48.391827071 +0000 UTC m=+841.408491787" watchObservedRunningTime="2026-10-31 00:16:48.392116233 +0000 UTC m=+841.408780944"
Oct 31 00:16:50.264184 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:50.264156    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:16:54.288166 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:54.288124    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:16:55.889601 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.889571    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"]
Oct 31 00:16:55.895294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.895266    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:55.955406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.950888    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-15eab294-serving-cert\""
Oct 31 00:16:55.993648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.993613    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:55.993788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.993680    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:55.993846 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.993786    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pl8q\" (UniqueName: \"kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:55.993888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:55.993856    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.002764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.002739    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"]
Oct 31 00:16:56.094848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.094813    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.095023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.094858    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pl8q\" (UniqueName: \"kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.095023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.094919    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.095140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.095029    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.095140 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:56.095110    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-serving-cert: secret "maas-api-e2e-ait-15eab294-serving-cert" not found
Oct 31 00:16:56.095242 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:16:56.095187    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls podName:04cce012-ee1f-4910-a0f8-76d6e1d00118 nodeName:}" failed. No retries permitted until 2026-10-31 00:16:56.595165829 +0000 UTC m=+849.611830528 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls") pod "maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" (UID: "04cce012-ee1f-4910-a0f8-76d6e1d00118") : secret "maas-api-e2e-ait-15eab294-serving-cert" not found
Oct 31 00:16:56.095508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.095487    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.097737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.097717    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.194271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.194233    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pl8q\" (UniqueName: \"kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.270653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.270625    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:16:56.600195 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.600112    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.602779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.602758    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\") pod \"maas-api-e2e-ait-15eab294-59cbb44dd7-9h299\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:56.826480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:56.826447    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:57.250406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:57.233899    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"]
Oct 31 00:16:57.259780 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:16:57.259751    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cce012_ee1f_4910_a0f8_76d6e1d00118.slice/crio-8981b64a9789f2f7c240a030351464ff72ce25eef14c823b0069280664b0696f WatchSource:0}: Error finding container 8981b64a9789f2f7c240a030351464ff72ce25eef14c823b0069280664b0696f: Status 404 returned error can't find the container with id 8981b64a9789f2f7c240a030351464ff72ce25eef14c823b0069280664b0696f
Oct 31 00:16:57.325999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:57.325967    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" event={"ID":"04cce012-ee1f-4910-a0f8-76d6e1d00118","Type":"ContainerStarted","Data":"8981b64a9789f2f7c240a030351464ff72ce25eef14c823b0069280664b0696f"}
Oct 31 00:16:59.361984 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:59.361937    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" event={"ID":"04cce012-ee1f-4910-a0f8-76d6e1d00118","Type":"ContainerStarted","Data":"534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f"}
Oct 31 00:16:59.362536 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:59.362033    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:16:59.481584 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:16:59.481527    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" podStartSLOduration=3.462285501 podStartE2EDuration="4.481512575s" totalImagesPullingTime="1.019227074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:59.480766439 +0000 UTC m=+852.497431175" watchObservedRunningTime="2026-10-31 00:16:59.481512575 +0000 UTC m=+852.498177288"
Oct 31 00:17:00.786075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.786044    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"]
Oct 31 00:17:00.789938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.789920    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.842147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.842115    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b83a178e-serving-cert\""
Oct 31 00:17:00.861347 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.861318    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.861547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.861523    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwt4\" (UniqueName: \"kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.861630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.861564    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.861630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.861612    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.894832 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.894766    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"]
Oct 31 00:17:00.962683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.962653    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwt4\" (UniqueName: \"kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.962856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.962694    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.962856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.962722    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.962856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.962809    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.963025 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:00.962941    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-serving-cert: secret "maas-api-e2e-derive-b83a178e-serving-cert" not found
Oct 31 00:17:00.963025 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:00.963016    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls podName:b8b8988f-68ed-4b40-ab17-213440ed3cdd nodeName:}" failed. No retries permitted until 2026-10-31 00:17:01.462996241 +0000 UTC m=+854.479660946 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls") pod "maas-api-e2e-derive-b83a178e-d67749c6-svbjg" (UID: "b8b8988f-68ed-4b40-ab17-213440ed3cdd") : secret "maas-api-e2e-derive-b83a178e-serving-cert" not found
Oct 31 00:17:00.963668 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.963648    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:00.965396 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:00.965359    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:01.024188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:01.024157    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwt4\" (UniqueName: \"kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:01.469747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:01.469703    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:01.472359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:01.472333    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\") pod \"maas-api-e2e-derive-b83a178e-d67749c6-svbjg\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:01.702940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:01.702896    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:02.082736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:02.082647    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"]
Oct 31 00:17:02.086661 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:17:02.086630    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b8988f_68ed_4b40_ab17_213440ed3cdd.slice/crio-18c725bd00743f0e236d53139f20b826cd24f012186f65cc548fffae422ace22 WatchSource:0}: Error finding container 18c725bd00743f0e236d53139f20b826cd24f012186f65cc548fffae422ace22: Status 404 returned error can't find the container with id 18c725bd00743f0e236d53139f20b826cd24f012186f65cc548fffae422ace22
Oct 31 00:17:02.224540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:02.224504    4478 patch_prober.go:29] interesting pod/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:17:02.224698 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:02.224571    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:17:02.378606 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:02.378529    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg" event={"ID":"b8b8988f-68ed-4b40-ab17-213440ed3cdd","Type":"ContainerStarted","Data":"18c725bd00743f0e236d53139f20b826cd24f012186f65cc548fffae422ace22"}
Oct 31 00:17:03.387214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:03.387113    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg" event={"ID":"b8b8988f-68ed-4b40-ab17-213440ed3cdd","Type":"ContainerStarted","Data":"c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e"}
Oct 31 00:17:03.387703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:03.387583    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:03.508701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:03.508642    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg" podStartSLOduration=2.525815877 podStartE2EDuration="3.508628559s" totalImagesPullingTime="982.812682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:03.507435172 +0000 UTC m=+856.524099909" watchObservedRunningTime="2026-10-31 00:17:03.508628559 +0000 UTC m=+856.525293271"
Oct 31 00:17:04.392983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:04.392951    4478 generic.go:356] "Generic (PLEG): container finished" podID="c503cfa5-d727-475b-a32c-df9626426349" containerID="18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8" exitCode=0
Oct 31 00:17:04.393438 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:04.393028    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" event={"ID":"c503cfa5-d727-475b-a32c-df9626426349","Type":"ContainerDied","Data":"18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"}
Oct 31 00:17:04.393486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:04.393458    4478 scope.go:122] "RemoveContainer" containerID="18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"
Oct 31 00:17:04.394668 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:04.394643    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:17:04.394746 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:04.394704    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:17:04.394746 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:04.394733    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:17:05.373547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:05.373520    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:17:05.400794 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:05.400762    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" event={"ID":"c503cfa5-d727-475b-a32c-df9626426349","Type":"ContainerStarted","Data":"b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a"}
Oct 31 00:17:06.150264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:06.149755    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:17:06.150264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:06.149874    4478 patch_prober.go:29] interesting pod/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:17:06.150264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:06.149929    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 31 00:17:07.150406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:07.141485    4478 patch_prober.go:29] interesting pod/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:17:07.150406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:07.141546    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 31 00:17:08.131781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:08.131747    4478 patch_prober.go:29] interesting pod/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:17:08.132091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:08.132062    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 31 00:17:09.131546 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:09.131516    4478 patch_prober.go:29] interesting pod/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:17:09.156748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:09.156695    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 31 00:17:09.398622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:09.398533    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:17:10.138508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:10.138472    4478 patch_prober.go:29] interesting pod/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body=
Oct 31 00:17:10.139108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:10.139076    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused"
Oct 31 00:17:11.135469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:11.135442    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:17:11.135734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:11.135716    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:17:11.136290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:11.136272    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:17:14.757597 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:14.757561    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:17:30.500153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:30.500116    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"]
Oct 31 00:17:30.605501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:30.605468    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"]
Oct 31 00:17:35.356749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.356700    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"]
Oct 31 00:17:35.501017 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.500982    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q" podUID="c794ec8b-c7d0-4e0c-a7be-deaf365920ec" containerName="payload-processing" containerID="cri-o://847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452" gracePeriod=25
Oct 31 00:17:35.606104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.606041    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t" podUID="46c87b23-99d2-4350-9743-7f60f739dfa8" containerName="payload-pre-processing" containerID="cri-o://e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d" gracePeriod=25
Oct 31 00:17:35.803769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.803745    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:17:35.850921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.850895    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:17:35.855994 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.855969    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"]
Oct 31 00:17:35.856171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.856151    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" podUID="a94ef019-4b6b-4f1b-afe9-18ea5120fb42" containerName="maas-api" containerID="cri-o://c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d" gracePeriod=30
Oct 31 00:17:35.922376 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.922347    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume\") pod \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") "
Oct 31 00:17:35.922558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.922472    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume\") pod \"46c87b23-99d2-4350-9743-7f60f739dfa8\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") "
Oct 31 00:17:35.922558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.922512    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr\" (UniqueName: \"kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr\") pod \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\" (UID: \"c794ec8b-c7d0-4e0c-a7be-deaf365920ec\") "
Oct 31 00:17:35.922653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.922639    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg\" (UniqueName: \"kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg\") pod \"46c87b23-99d2-4350-9743-7f60f739dfa8\" (UID: \"46c87b23-99d2-4350-9743-7f60f739dfa8\") "
Oct 31 00:17:35.922826 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.922789    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume" pod "c794ec8b-c7d0-4e0c-a7be-deaf365920ec" (UID: "c794ec8b-c7d0-4e0c-a7be-deaf365920ec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:35.923026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.923007    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:35.923519 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.923423    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume" pod "46c87b23-99d2-4350-9743-7f60f739dfa8" (UID: "46c87b23-99d2-4350-9743-7f60f739dfa8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:35.925116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.925089    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg" pod "46c87b23-99d2-4350-9743-7f60f739dfa8" (UID: "46c87b23-99d2-4350-9743-7f60f739dfa8"). InnerVolumeSpecName "kube-api-access-qwchg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:35.925116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:35.925102    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr" pod "c794ec8b-c7d0-4e0c-a7be-deaf365920ec" (UID: "c794ec8b-c7d0-4e0c-a7be-deaf365920ec"). InnerVolumeSpecName "kube-api-access-hhhvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:36.024370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.024334    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46c87b23-99d2-4350-9743-7f60f739dfa8-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.024370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.024364    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhhvr\" (UniqueName: \"kubernetes.io/projected/c794ec8b-c7d0-4e0c-a7be-deaf365920ec-kube-api-access-hhhvr\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.024370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.024373    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwchg\" (UniqueName: \"kubernetes.io/projected/46c87b23-99d2-4350-9743-7f60f739dfa8-kube-api-access-qwchg\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.100455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.100433    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:17:36.236896 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.236860    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs\") pod \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") "
Oct 31 00:17:36.237277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.237252    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\") pod \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") "
Oct 31 00:17:36.237460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.237441    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm\" (UniqueName: \"kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm\") pod \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") "
Oct 31 00:17:36.237590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.237576    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca\") pod \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\" (UID: \"a94ef019-4b6b-4f1b-afe9-18ea5120fb42\") "
Oct 31 00:17:36.238820 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.238784    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca" pod "a94ef019-4b6b-4f1b-afe9-18ea5120fb42" (UID: "a94ef019-4b6b-4f1b-afe9-18ea5120fb42"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:36.241675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.241648    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs" pod "a94ef019-4b6b-4f1b-afe9-18ea5120fb42" (UID: "a94ef019-4b6b-4f1b-afe9-18ea5120fb42"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:36.242772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.242751    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls" pod "a94ef019-4b6b-4f1b-afe9-18ea5120fb42" (UID: "a94ef019-4b6b-4f1b-afe9-18ea5120fb42"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:36.244937 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.244912    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm" pod "a94ef019-4b6b-4f1b-afe9-18ea5120fb42" (UID: "a94ef019-4b6b-4f1b-afe9-18ea5120fb42"). InnerVolumeSpecName "kube-api-access-cg8zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:36.339264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.339230    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.339264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.339260    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cg8zm\" (UniqueName: \"kubernetes.io/projected/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-kube-api-access-cg8zm\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.339264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.339270    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.339544 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.339280    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a94ef019-4b6b-4f1b-afe9-18ea5120fb42-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:36.556041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.556010    4478 generic.go:356] "Generic (PLEG): container finished" podID="c794ec8b-c7d0-4e0c-a7be-deaf365920ec" containerID="847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452" exitCode=0
Oct 31 00:17:36.556427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.556052    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"
Oct 31 00:17:36.556427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.556085    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q" event={"ID":"c794ec8b-c7d0-4e0c-a7be-deaf365920ec","Type":"ContainerDied","Data":"847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452"}
Oct 31 00:17:36.556427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.556137    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q" event={"ID":"c794ec8b-c7d0-4e0c-a7be-deaf365920ec","Type":"ContainerDied","Data":"d4be8f59d1cd841875cd560842f914f3d7a4c9d69a574a5fa7ef91de5b88d8c3"}
Oct 31 00:17:36.556427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.556164    4478 scope.go:122] "RemoveContainer" containerID="847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452"
Oct 31 00:17:36.557517 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.557494    4478 generic.go:356] "Generic (PLEG): container finished" podID="46c87b23-99d2-4350-9743-7f60f739dfa8" containerID="e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d" exitCode=0
Oct 31 00:17:36.557764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.557546    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"
Oct 31 00:17:36.557764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.557574    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t" event={"ID":"46c87b23-99d2-4350-9743-7f60f739dfa8","Type":"ContainerDied","Data":"e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d"}
Oct 31 00:17:36.557764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.557629    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t" event={"ID":"46c87b23-99d2-4350-9743-7f60f739dfa8","Type":"ContainerDied","Data":"6909d0015c6b1dbbc6f664155987138a7901ab0c47e0867e82d67158e5ed692c"}
Oct 31 00:17:36.559056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.559037    4478 generic.go:356] "Generic (PLEG): container finished" podID="a94ef019-4b6b-4f1b-afe9-18ea5120fb42" containerID="c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d" exitCode=0
Oct 31 00:17:36.559187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.559075    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"
Oct 31 00:17:36.559187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.559087    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" event={"ID":"a94ef019-4b6b-4f1b-afe9-18ea5120fb42","Type":"ContainerDied","Data":"c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d"}
Oct 31 00:17:36.559187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.559126    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj" event={"ID":"a94ef019-4b6b-4f1b-afe9-18ea5120fb42","Type":"ContainerDied","Data":"847a6c9581617d58c8c090a27f031a78a2b3337f33ed088379f3d02f453488b7"}
Oct 31 00:17:36.565060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.565042    4478 scope.go:122] "RemoveContainer" containerID="847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452"
Oct 31 00:17:36.565345 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:36.565326    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452\": container with ID starting with 847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452 not found: ID does not exist" containerID="847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452"
Oct 31 00:17:36.565434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.565351    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452"} err="failed to get container status \"847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452\": rpc error: code = NotFound desc = could not find container \"847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452\": container with ID starting with 847965489d9fe1dce72c8a2d9c574334af3b6470848c4fe74931cc34c8164452 not found: ID does not exist"
Oct 31 00:17:36.565434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.565367    4478 scope.go:122] "RemoveContainer" containerID="e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d"
Oct 31 00:17:36.576184 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.576147    4478 scope.go:122] "RemoveContainer" containerID="e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d"
Oct 31 00:17:36.576478 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:36.576456    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d\": container with ID starting with e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d not found: ID does not exist" containerID="e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d"
Oct 31 00:17:36.576544 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.576495    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d"} err="failed to get container status \"e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d\": rpc error: code = NotFound desc = could not find container \"e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d\": container with ID starting with e6c6a1b3f4dc2d56bad1f4b95bd2ca49fca8154f62136af8a609e60737b49d1d not found: ID does not exist"
Oct 31 00:17:36.576544 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.576513    4478 scope.go:122] "RemoveContainer" containerID="c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d"
Oct 31 00:17:36.584065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.584042    4478 scope.go:122] "RemoveContainer" containerID="c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d"
Oct 31 00:17:36.584322 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:36.584301    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d\": container with ID starting with c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d not found: ID does not exist" containerID="c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d"
Oct 31 00:17:36.584408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.584331    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d"} err="failed to get container status \"c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d\": rpc error: code = NotFound desc = could not find container \"c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d\": container with ID starting with c67d57a7cda53832aeaba0471f7e9c1e8fb39937a746a73ac3b3a3a82e1b2f5d not found: ID does not exist"
Oct 31 00:17:36.720547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.720508    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"]
Oct 31 00:17:36.725676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.725649    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4b78c516-777fcd7fff-wr46t"]
Oct 31 00:17:36.834973 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.834906    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"]
Oct 31 00:17:36.839984 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.839931    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e1bbe4b-7cf55f6bfc-r6bxj"]
Oct 31 00:17:36.954358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.954327    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"]
Oct 31 00:17:36.960510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:36.960485    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-4b78c516-67786978cc-ps68q"]
Oct 31 00:17:37.044057 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.044024    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"]
Oct 31 00:17:37.044301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.044264    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" podUID="b74374be-0eed-4bba-8059-bd563271eeef" containerName="maas-api" containerID="cri-o://38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117" gracePeriod=30
Oct 31 00:17:37.117542 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.117462    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c87b23-99d2-4350-9743-7f60f739dfa8" path="/var/lib/kubelet/pods/46c87b23-99d2-4350-9743-7f60f739dfa8/volumes"
Oct 31 00:17:37.118058 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.118033    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a94ef019-4b6b-4f1b-afe9-18ea5120fb42" path="/var/lib/kubelet/pods/a94ef019-4b6b-4f1b-afe9-18ea5120fb42/volumes"
Oct 31 00:17:37.118643 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.118621    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c794ec8b-c7d0-4e0c-a7be-deaf365920ec" path="/var/lib/kubelet/pods/c794ec8b-c7d0-4e0c-a7be-deaf365920ec/volumes"
Oct 31 00:17:37.313832 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.313807    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:17:37.459282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.458186    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca\") pod \"b74374be-0eed-4bba-8059-bd563271eeef\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") "
Oct 31 00:17:37.459282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.458248    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9\" (UniqueName: \"kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9\") pod \"b74374be-0eed-4bba-8059-bd563271eeef\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") "
Oct 31 00:17:37.459282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.458299    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs\") pod \"b74374be-0eed-4bba-8059-bd563271eeef\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") "
Oct 31 00:17:37.459282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.458433    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\") pod \"b74374be-0eed-4bba-8059-bd563271eeef\" (UID: \"b74374be-0eed-4bba-8059-bd563271eeef\") "
Oct 31 00:17:37.459739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.459708    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca" pod "b74374be-0eed-4bba-8059-bd563271eeef" (UID: "b74374be-0eed-4bba-8059-bd563271eeef"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:37.473273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.473232    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls" pod "b74374be-0eed-4bba-8059-bd563271eeef" (UID: "b74374be-0eed-4bba-8059-bd563271eeef"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:37.473518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.473462    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs" pod "b74374be-0eed-4bba-8059-bd563271eeef" (UID: "b74374be-0eed-4bba-8059-bd563271eeef"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:37.473609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.473588    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9" pod "b74374be-0eed-4bba-8059-bd563271eeef" (UID: "b74374be-0eed-4bba-8059-bd563271eeef"). InnerVolumeSpecName "kube-api-access-xgvc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:37.559817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.559778    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b74374be-0eed-4bba-8059-bd563271eeef-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.559817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.559816    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgvc9\" (UniqueName: \"kubernetes.io/projected/b74374be-0eed-4bba-8059-bd563271eeef-kube-api-access-xgvc9\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.559817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.559835    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.559817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.559848    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b74374be-0eed-4bba-8059-bd563271eeef-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:37.568566 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.568542    4478 generic.go:356] "Generic (PLEG): container finished" podID="b74374be-0eed-4bba-8059-bd563271eeef" containerID="38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117" exitCode=0
Oct 31 00:17:37.568689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.568593    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"
Oct 31 00:17:37.568689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.568619    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" event={"ID":"b74374be-0eed-4bba-8059-bd563271eeef","Type":"ContainerDied","Data":"38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117"}
Oct 31 00:17:37.568689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.568664    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv" event={"ID":"b74374be-0eed-4bba-8059-bd563271eeef","Type":"ContainerDied","Data":"e6e5d6dfd3548704f28bc2fa9ddaaf106c1481557486d3abff3bf472a4bf45ac"}
Oct 31 00:17:37.568835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.568693    4478 scope.go:122] "RemoveContainer" containerID="38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117"
Oct 31 00:17:37.578277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.578256    4478 scope.go:122] "RemoveContainer" containerID="38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117"
Oct 31 00:17:37.578567 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:37.578549    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117\": container with ID starting with 38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117 not found: ID does not exist" containerID="38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117"
Oct 31 00:17:37.578648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.578574    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117"} err="failed to get container status \"38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117\": rpc error: code = NotFound desc = could not find container \"38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117\": container with ID starting with 38f3f8d283afbf3235965d27ea0951d5894560656e8ec72dcd05a57faa998117 not found: ID does not exist"
Oct 31 00:17:37.732099 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.732025    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"]
Oct 31 00:17:37.739137 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:37.739108    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4b78c516-675dcff4fc-227xv"]
Oct 31 00:17:38.252769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.252736    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"]
Oct 31 00:17:38.252948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.252929    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d" podUID="942ec4be-d10a-4c44-ae69-2495548e4330" containerName="maas-api" containerID="cri-o://4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93" gracePeriod=30
Oct 31 00:17:38.501090 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.501067    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:17:38.574667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.574598    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z\" (UniqueName: \"kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z\") pod \"942ec4be-d10a-4c44-ae69-2495548e4330\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") "
Oct 31 00:17:38.575081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.574681    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls\") pod \"942ec4be-d10a-4c44-ae69-2495548e4330\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") "
Oct 31 00:17:38.575081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.574746    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca\") pod \"942ec4be-d10a-4c44-ae69-2495548e4330\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") "
Oct 31 00:17:38.575081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.574783    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs\") pod \"942ec4be-d10a-4c44-ae69-2495548e4330\" (UID: \"942ec4be-d10a-4c44-ae69-2495548e4330\") "
Oct 31 00:17:38.575213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.575176    4478 generic.go:356] "Generic (PLEG): container finished" podID="942ec4be-d10a-4c44-ae69-2495548e4330" containerID="4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93" exitCode=0
Oct 31 00:17:38.575251 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.575227    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"
Oct 31 00:17:38.575302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.575272    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d" event={"ID":"942ec4be-d10a-4c44-ae69-2495548e4330","Type":"ContainerDied","Data":"4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93"}
Oct 31 00:17:38.575361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.575343    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d" event={"ID":"942ec4be-d10a-4c44-ae69-2495548e4330","Type":"ContainerDied","Data":"5e9dc56ecea535f94eb65cd1e1cc76f03f0256eb87cdbc8390850d157a64152f"}
Oct 31 00:17:38.575443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.575373    4478 scope.go:122] "RemoveContainer" containerID="4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93"
Oct 31 00:17:38.575610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.575584    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca" pod "942ec4be-d10a-4c44-ae69-2495548e4330" (UID: "942ec4be-d10a-4c44-ae69-2495548e4330"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:38.577283 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.577254    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs" pod "942ec4be-d10a-4c44-ae69-2495548e4330" (UID: "942ec4be-d10a-4c44-ae69-2495548e4330"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:38.577479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.577458    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z" pod "942ec4be-d10a-4c44-ae69-2495548e4330" (UID: "942ec4be-d10a-4c44-ae69-2495548e4330"). InnerVolumeSpecName "kube-api-access-d2f8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:38.577588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.577570    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls" pod "942ec4be-d10a-4c44-ae69-2495548e4330" (UID: "942ec4be-d10a-4c44-ae69-2495548e4330"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:17:38.594756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.594740    4478 scope.go:122] "RemoveContainer" containerID="4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93"
Oct 31 00:17:38.595061 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:38.595029    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93\": container with ID starting with 4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93 not found: ID does not exist" containerID="4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93"
Oct 31 00:17:38.595123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.595075    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93"} err="failed to get container status \"4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93\": rpc error: code = NotFound desc = could not find container \"4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93\": container with ID starting with 4d84b2630518b8e5dc4c352c3105169e62c80794408b953c8ebb37f2771dcb93 not found: ID does not exist"
Oct 31 00:17:38.676357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.676324    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2f8z\" (UniqueName: \"kubernetes.io/projected/942ec4be-d10a-4c44-ae69-2495548e4330-kube-api-access-d2f8z\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:38.676357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.676351    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:38.676357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.676361    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/942ec4be-d10a-4c44-ae69-2495548e4330-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:38.676650 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.676369    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/942ec4be-d10a-4c44-ae69-2495548e4330-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:38.999628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:38.999598    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"]
Oct 31 00:17:39.002559 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:39.002534    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ab60e97-597b6d8944-kr72d"]
Oct 31 00:17:39.117070 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:39.117040    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942ec4be-d10a-4c44-ae69-2495548e4330" path="/var/lib/kubelet/pods/942ec4be-d10a-4c44-ae69-2495548e4330/volumes"
Oct 31 00:17:39.117448 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:39.117435    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b74374be-0eed-4bba-8059-bd563271eeef" path="/var/lib/kubelet/pods/b74374be-0eed-4bba-8059-bd563271eeef/volumes"
Oct 31 00:17:40.358004 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.357960    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh" podUID="9d778103-d6a8-4f52-8848-c28d4c6c4955" containerName="payload-pre-processing" containerID="cri-o://124bf7a104c55c634e52723c69165b88ffd6c237760f1c6fc4adad8a6e62fdcf" gracePeriod=25
Oct 31 00:17:40.589607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.589575    4478 generic.go:356] "Generic (PLEG): container finished" podID="9d778103-d6a8-4f52-8848-c28d4c6c4955" containerID="124bf7a104c55c634e52723c69165b88ffd6c237760f1c6fc4adad8a6e62fdcf" exitCode=0
Oct 31 00:17:40.589787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.589650    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh" event={"ID":"9d778103-d6a8-4f52-8848-c28d4c6c4955","Type":"ContainerDied","Data":"124bf7a104c55c634e52723c69165b88ffd6c237760f1c6fc4adad8a6e62fdcf"}
Oct 31 00:17:40.614828 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.614768    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:17:40.696094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.696058    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p\" (UniqueName: \"kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p\") pod \"9d778103-d6a8-4f52-8848-c28d4c6c4955\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") "
Oct 31 00:17:40.696281 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.696159    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume\") pod \"9d778103-d6a8-4f52-8848-c28d4c6c4955\" (UID: \"9d778103-d6a8-4f52-8848-c28d4c6c4955\") "
Oct 31 00:17:40.696525 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.696502    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume" pod "9d778103-d6a8-4f52-8848-c28d4c6c4955" (UID: "9d778103-d6a8-4f52-8848-c28d4c6c4955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:40.698397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.698352    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p" pod "9d778103-d6a8-4f52-8848-c28d4c6c4955" (UID: "9d778103-d6a8-4f52-8848-c28d4c6c4955"). InnerVolumeSpecName "kube-api-access-cgd2p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:40.797439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.797412    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d778103-d6a8-4f52-8848-c28d4c6c4955-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:40.797439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:40.797436    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgd2p\" (UniqueName: \"kubernetes.io/projected/9d778103-d6a8-4f52-8848-c28d4c6c4955-kube-api-access-cgd2p\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:41.353145 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.353109    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"]
Oct 31 00:17:41.353416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.353368    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy" containerID="cri-o://b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a" gracePeriod=30
Oct 31 00:17:41.497999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.497966    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"]
Oct 31 00:17:41.498540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498518    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b74374be-0eed-4bba-8059-bd563271eeef" containerName="maas-api"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498543    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b74374be-0eed-4bba-8059-bd563271eeef" containerName="maas-api"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498601    4478 cpu_manager.go:414] "containerMap: removing container" podUID="9d778103-d6a8-4f52-8848-c28d4c6c4955" containerName="payload-pre-processing"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498611    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d778103-d6a8-4f52-8848-c28d4c6c4955" containerName="payload-pre-processing"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498641    4478 cpu_manager.go:414] "containerMap: removing container" podUID="46c87b23-99d2-4350-9743-7f60f739dfa8" containerName="payload-pre-processing"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498652    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c87b23-99d2-4350-9743-7f60f739dfa8" containerName="payload-pre-processing"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498665    4478 cpu_manager.go:414] "containerMap: removing container" podUID="942ec4be-d10a-4c44-ae69-2495548e4330" containerName="maas-api"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498672    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942ec4be-d10a-4c44-ae69-2495548e4330" containerName="maas-api"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498688    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a94ef019-4b6b-4f1b-afe9-18ea5120fb42" containerName="maas-api"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498696    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a94ef019-4b6b-4f1b-afe9-18ea5120fb42" containerName="maas-api"
Oct 31 00:17:41.498704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498707    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c794ec8b-c7d0-4e0c-a7be-deaf365920ec" containerName="payload-processing"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498713    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c794ec8b-c7d0-4e0c-a7be-deaf365920ec" containerName="payload-processing"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498793    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a94ef019-4b6b-4f1b-afe9-18ea5120fb42" containerName="maas-api"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498813    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c794ec8b-c7d0-4e0c-a7be-deaf365920ec" containerName="payload-processing"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498825    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b74374be-0eed-4bba-8059-bd563271eeef" containerName="maas-api"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498836    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d778103-d6a8-4f52-8848-c28d4c6c4955" containerName="payload-pre-processing"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498847    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c87b23-99d2-4350-9743-7f60f739dfa8" containerName="payload-pre-processing"
Oct 31 00:17:41.499290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.498855    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="942ec4be-d10a-4c44-ae69-2495548e4330" containerName="maas-api"
Oct 31 00:17:41.501739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.501721    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.553693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.553667    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-7abb95-openshift-default-dockercfg-d8tvz\""
Oct 31 00:17:41.562963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.562931    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"]
Oct 31 00:17:41.596858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.596828    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh" event={"ID":"9d778103-d6a8-4f52-8848-c28d4c6c4955","Type":"ContainerDied","Data":"2ad5949569bbcd2af9d7ccedaa680256089e1e3134ab0dbab65ac293ffeef05f"}
Oct 31 00:17:41.597007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.596892    4478 scope.go:122] "RemoveContainer" containerID="124bf7a104c55c634e52723c69165b88ffd6c237760f1c6fc4adad8a6e62fdcf"
Oct 31 00:17:41.597007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.596835    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"
Oct 31 00:17:41.612481 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612460    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxzlp\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612491    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612516    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612642    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612726    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612791 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612777    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612803    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612852    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.612943 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.612932    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714011    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxzlp\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714227 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714050    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714227 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714074    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714227 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714196    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714238    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714275    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714295    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714317    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714337    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714615    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714814    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714919 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714840    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714919 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714847    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.714919 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.714905    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.716710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.716689    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.716793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.716771    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.731265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.731244    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"]
Oct 31 00:17:41.738148 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.738125    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6ab60e97-6695c98759-czvkh"]
Oct 31 00:17:41.774855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.774781    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.774959 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.774945    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxzlp\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:41.813305 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:41.813282    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:42.250734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.250696    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"]
Oct 31 00:17:42.253811 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:17:42.253782    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2e42da_51b1_40a8_bbf9_56d1bdbd5dc8.slice/crio-6cf0f53c2bb9415417f1c3fff2fb4b0fea593e71ff6cd4b712bfb4a3e04fa5c1 WatchSource:0}: Error finding container 6cf0f53c2bb9415417f1c3fff2fb4b0fea593e71ff6cd4b712bfb4a3e04fa5c1: Status 404 returned error can't find the container with id 6cf0f53c2bb9415417f1c3fff2fb4b0fea593e71ff6cd4b712bfb4a3e04fa5c1
Oct 31 00:17:42.256192 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.256154    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:17:42.256279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.256240    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:17:42.256328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.256279    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:17:42.604885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.604785    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" event={"ID":"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8","Type":"ContainerStarted","Data":"167c390049597f8c482fed6eace24d598b98e7b93d709bdf535566f612909871"}
Oct 31 00:17:42.604885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.604850    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" event={"ID":"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8","Type":"ContainerStarted","Data":"6cf0f53c2bb9415417f1c3fff2fb4b0fea593e71ff6cd4b712bfb4a3e04fa5c1"}
Oct 31 00:17:42.734854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.734783    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podStartSLOduration=1.734764777 podStartE2EDuration="1.734764777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:42.732136471 +0000 UTC m=+895.748801181" watchObservedRunningTime="2026-10-31 00:17:42.734764777 +0000 UTC m=+895.751429490"
Oct 31 00:17:42.813430 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:42.813401    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:43.116514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:43.116484    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d778103-d6a8-4f52-8848-c28d4c6c4955" path="/var/lib/kubelet/pods/9d778103-d6a8-4f52-8848-c28d4c6c4955/volumes"
Oct 31 00:17:43.819182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:43.819145    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:44.613770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:44.613744    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:44.614686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:44.614668    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:17:44.758424 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:44.758359    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:17:46.602349 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.602326    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:17:46.623161 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.623095    4478 generic.go:356] "Generic (PLEG): container finished" podID="c503cfa5-d727-475b-a32c-df9626426349" containerID="b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a" exitCode=0
Oct 31 00:17:46.623161 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.623136    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"
Oct 31 00:17:46.623331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.623182    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" event={"ID":"c503cfa5-d727-475b-a32c-df9626426349","Type":"ContainerDied","Data":"b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a"}
Oct 31 00:17:46.623331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.623231    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl" event={"ID":"c503cfa5-d727-475b-a32c-df9626426349","Type":"ContainerDied","Data":"b58c832722add47851a88e2ed87e9b7bdbb960837576f0c618ec82682690e6d8"}
Oct 31 00:17:46.623331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.623253    4478 scope.go:122] "RemoveContainer" containerID="b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a"
Oct 31 00:17:46.631527 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.631511    4478 scope.go:122] "RemoveContainer" containerID="18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"
Oct 31 00:17:46.639087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.639072    4478 scope.go:122] "RemoveContainer" containerID="b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a"
Oct 31 00:17:46.639322 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:46.639307    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a\": container with ID starting with b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a not found: ID does not exist" containerID="b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a"
Oct 31 00:17:46.639359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.639330    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a"} err="failed to get container status \"b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a\": rpc error: code = NotFound desc = could not find container \"b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a\": container with ID starting with b971f505f524450b4a3c97253c6fbbb0098db98ad1aa49ec35e27c3dddd4142a not found: ID does not exist"
Oct 31 00:17:46.639359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.639346    4478 scope.go:122] "RemoveContainer" containerID="18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"
Oct 31 00:17:46.639606 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:17:46.639591    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8\": container with ID starting with 18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8 not found: ID does not exist" containerID="18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"
Oct 31 00:17:46.639656 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.639622    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8"} err="failed to get container status \"18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8\": rpc error: code = NotFound desc = could not find container \"18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8\": container with ID starting with 18245a4194ef7c3d1de3bef6e9bcc519a3018e01824f6228ccdf24cc22e383a8 not found: ID does not exist"
Oct 31 00:17:46.763542 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763506    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763542 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763552    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763575    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763615    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763658    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763689    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763722    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763775    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763792    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:46.763835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763803    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket\") pod \"c503cfa5-d727-475b-a32c-df9626426349\" (UID: \"c503cfa5-d727-475b-a32c-df9626426349\") "
Oct 31 00:17:46.763835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763810    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:46.764118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.763929    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:46.764118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.764013    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:46.764294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.764272    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.764360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.764301    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.764360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.764317    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.764360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.764334    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.764541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.764365    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:17:46.766113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.766089    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:17:46.766352 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.766331    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:17:46.766463 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.766443    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:46.766535 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.766522    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs" pod "c503cfa5-d727-475b-a32c-df9626426349" (UID: "c503cfa5-d727-475b-a32c-df9626426349"). InnerVolumeSpecName "kube-api-access-msbbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:17:46.865431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.865405    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c503cfa5-d727-475b-a32c-df9626426349-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.865431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.865426    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c503cfa5-d727-475b-a32c-df9626426349-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.865591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.865437    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.865591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.865446    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c503cfa5-d727-475b-a32c-df9626426349-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:46.865591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:46.865455    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msbbs\" (UniqueName: \"kubernetes.io/projected/c503cfa5-d727-475b-a32c-df9626426349-kube-api-access-msbbs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:17:47.047457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:47.047426    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"]
Oct 31 00:17:47.049532 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:47.049510    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-6ab60e97-openshift-default-9d9dcc997-62bzl"]
Oct 31 00:17:47.117065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:17:47.117037    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c503cfa5-d727-475b-a32c-df9626426349" path="/var/lib/kubelet/pods/c503cfa5-d727-475b-a32c-df9626426349/volumes"
Oct 31 00:18:00.502774 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.502742    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"]
Oct 31 00:18:00.503196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.503108    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy"
Oct 31 00:18:00.503196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.503119    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy"
Oct 31 00:18:00.503196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.503131    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy"
Oct 31 00:18:00.503196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.503138    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy"
Oct 31 00:18:00.503335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.503213    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy"
Oct 31 00:18:00.503335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.503222    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c503cfa5-d727-475b-a32c-df9626426349" containerName="istio-proxy"
Oct 31 00:18:00.506093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.506076    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.570675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.570645    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-7abb95-serving-cert\""
Oct 31 00:18:00.612796 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.612767    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"]
Oct 31 00:18:00.697800 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.697772    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.697971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.697804    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.697971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.697938    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-59k2t\" (UniqueName: \"kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.698062 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.697999    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.799226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.799112    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-59k2t\" (UniqueName: \"kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.799226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.799200    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.799470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.799244    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.799470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.799273    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.799799 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:00.799769    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-serving-cert: secret "maas-api-e2e-worker-w3-7abb95-serving-cert" not found
Oct 31 00:18:00.799905 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:00.799848    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls podName:dac8e284-715e-4d78-8865-31818b6749d7 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:01.29983334 +0000 UTC m=+914.316498033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls") pod "maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" (UID: "dac8e284-715e-4d78-8865-31818b6749d7") : secret "maas-api-e2e-worker-w3-7abb95-serving-cert" not found
Oct 31 00:18:00.800234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.800203    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.802591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.802569    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:00.862439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:00.862414    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-59k2t\" (UniqueName: \"kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:01.304914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:01.304878    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:01.307430 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:01.307410    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\") pod \"maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:01.418100 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:01.418064    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:01.792736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:01.792700    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"]
Oct 31 00:18:01.803102 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:01.803074    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddac8e284_715e_4d78_8865_31818b6749d7.slice/crio-36838cb0b4dc2687a201b1dc645f1ec87f7a0ceb2e644dad11c6d46bc9fe42d0 WatchSource:0}: Error finding container 36838cb0b4dc2687a201b1dc645f1ec87f7a0ceb2e644dad11c6d46bc9fe42d0: Status 404 returned error can't find the container with id 36838cb0b4dc2687a201b1dc645f1ec87f7a0ceb2e644dad11c6d46bc9fe42d0
Oct 31 00:18:02.691797 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:02.691757    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" event={"ID":"dac8e284-715e-4d78-8865-31818b6749d7","Type":"ContainerStarted","Data":"36838cb0b4dc2687a201b1dc645f1ec87f7a0ceb2e644dad11c6d46bc9fe42d0"}
Oct 31 00:18:03.697677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:03.697640    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" event={"ID":"dac8e284-715e-4d78-8865-31818b6749d7","Type":"ContainerStarted","Data":"8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54"}
Oct 31 00:18:03.698141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:03.697821    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:03.823029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:03.822968    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" podStartSLOduration=2.748670061 podStartE2EDuration="3.822950445s" totalImagesPullingTime="1.074280384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:03.814494664 +0000 UTC m=+916.831159398" watchObservedRunningTime="2026-10-31 00:18:03.822950445 +0000 UTC m=+916.839615157"
Oct 31 00:18:05.384291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.384256    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"]
Oct 31 00:18:05.386705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.386685    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.437685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.437653    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8mcp\" (UniqueName: \"kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.437802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.437695    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.437802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.437777    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.437899 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.437878    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.439898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.439882    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-0144bd-serving-cert\""
Oct 31 00:18:05.491108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.491072    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"]
Oct 31 00:18:05.538830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.538786    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8mcp\" (UniqueName: \"kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.539042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.538845    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.539042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.538892    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.539042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.538972    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.539251 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:05.539068    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-serving-cert: secret "maas-api-e2e-worker-w4-0144bd-serving-cert" not found
Oct 31 00:18:05.539251 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:05.539162    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls podName:b68e8749-2183-4329-97c7-8f5c9ddb0fbb nodeName:}" failed. No retries permitted until 2026-10-31 00:18:06.039139652 +0000 UTC m=+919.055804370 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls") pod "maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" (UID: "b68e8749-2183-4329-97c7-8f5c9ddb0fbb") : secret "maas-api-e2e-worker-w4-0144bd-serving-cert" not found
Oct 31 00:18:05.539807 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.539782    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.542163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.542136    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:05.600260 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:05.600218    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8mcp\" (UniqueName: \"kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:06.046787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:06.046750    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:06.049372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:06.049347    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\") pod \"maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:06.298590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:06.298499    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:06.672582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:06.672557    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"]
Oct 31 00:18:06.674940 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:06.674907    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb68e8749_2183_4329_97c7_8f5c9ddb0fbb.slice/crio-d9aac23c96a3424415eb86e5a667c0ac01d3c56045062d25e7ebb7d0793d47d2 WatchSource:0}: Error finding container d9aac23c96a3424415eb86e5a667c0ac01d3c56045062d25e7ebb7d0793d47d2: Status 404 returned error can't find the container with id d9aac23c96a3424415eb86e5a667c0ac01d3c56045062d25e7ebb7d0793d47d2
Oct 31 00:18:06.712994 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:06.712969    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" event={"ID":"b68e8749-2183-4329-97c7-8f5c9ddb0fbb","Type":"ContainerStarted","Data":"d9aac23c96a3424415eb86e5a667c0ac01d3c56045062d25e7ebb7d0793d47d2"}
Oct 31 00:18:07.719038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:07.719006    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" event={"ID":"b68e8749-2183-4329-97c7-8f5c9ddb0fbb","Type":"ContainerStarted","Data":"01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c"}
Oct 31 00:18:07.719501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:07.719116    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:07.837034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:07.836920    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" podStartSLOduration=2.003499741 podStartE2EDuration="2.836902519s" totalImagesPullingTime="833.402778ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:07.836064726 +0000 UTC m=+920.852729492" watchObservedRunningTime="2026-10-31 00:18:07.836902519 +0000 UTC m=+920.853567232"
Oct 31 00:18:09.707850 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:09.707821    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 00:18:09.812275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:09.812243    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"]
Oct 31 00:18:09.812462 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:09.812439    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" podUID="04cce012-ee1f-4910-a0f8-76d6e1d00118" containerName="maas-api" containerID="cri-o://534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f" gracePeriod=30
Oct 31 00:18:10.114984 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.114959    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:18:10.192418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.192354    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\") pod \"04cce012-ee1f-4910-a0f8-76d6e1d00118\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") "
Oct 31 00:18:10.192609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.192452    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q\" (UniqueName: \"kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q\") pod \"04cce012-ee1f-4910-a0f8-76d6e1d00118\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") "
Oct 31 00:18:10.192609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.192567    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca\") pod \"04cce012-ee1f-4910-a0f8-76d6e1d00118\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") "
Oct 31 00:18:10.192731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.192624    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs\") pod \"04cce012-ee1f-4910-a0f8-76d6e1d00118\" (UID: \"04cce012-ee1f-4910-a0f8-76d6e1d00118\") "
Oct 31 00:18:10.193783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.193688    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca" pod "04cce012-ee1f-4910-a0f8-76d6e1d00118" (UID: "04cce012-ee1f-4910-a0f8-76d6e1d00118"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:10.196686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.196602    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls" pod "04cce012-ee1f-4910-a0f8-76d6e1d00118" (UID: "04cce012-ee1f-4910-a0f8-76d6e1d00118"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:10.196973 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.196946    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q" pod "04cce012-ee1f-4910-a0f8-76d6e1d00118" (UID: "04cce012-ee1f-4910-a0f8-76d6e1d00118"). InnerVolumeSpecName "kube-api-access-4pl8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:10.197152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.197130    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs" pod "04cce012-ee1f-4910-a0f8-76d6e1d00118" (UID: "04cce012-ee1f-4910-a0f8-76d6e1d00118"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:10.293434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.293332    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04cce012-ee1f-4910-a0f8-76d6e1d00118-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:10.293434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.293359    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:10.293434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.293369    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04cce012-ee1f-4910-a0f8-76d6e1d00118-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:10.293434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.293405    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pl8q\" (UniqueName: \"kubernetes.io/projected/04cce012-ee1f-4910-a0f8-76d6e1d00118-kube-api-access-4pl8q\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:10.741527 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.741490    4478 generic.go:356] "Generic (PLEG): container finished" podID="04cce012-ee1f-4910-a0f8-76d6e1d00118" containerID="534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f" exitCode=0
Oct 31 00:18:10.742204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.742174    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" event={"ID":"04cce012-ee1f-4910-a0f8-76d6e1d00118","Type":"ContainerDied","Data":"534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f"}
Oct 31 00:18:10.742373 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.742355    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299" event={"ID":"04cce012-ee1f-4910-a0f8-76d6e1d00118","Type":"ContainerDied","Data":"8981b64a9789f2f7c240a030351464ff72ce25eef14c823b0069280664b0696f"}
Oct 31 00:18:10.742537 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.742522    4478 scope.go:122] "RemoveContainer" containerID="534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f"
Oct 31 00:18:10.742789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.742773    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"
Oct 31 00:18:10.777637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.777618    4478 scope.go:122] "RemoveContainer" containerID="534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f"
Oct 31 00:18:10.778094 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:10.778072    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f\": container with ID starting with 534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f not found: ID does not exist" containerID="534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f"
Oct 31 00:18:10.778157 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.778119    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f"} err="failed to get container status \"534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f\": rpc error: code = NotFound desc = could not find container \"534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f\": container with ID starting with 534446e7faf2bcc6498236dee528b7d3c7cb42b308032756a6a98067af00142f not found: ID does not exist"
Oct 31 00:18:10.913646 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.913612    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"]
Oct 31 00:18:10.917066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:10.917042    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-15eab294-59cbb44dd7-9h299"]
Oct 31 00:18:11.116265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.116185    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04cce012-ee1f-4910-a0f8-76d6e1d00118" path="/var/lib/kubelet/pods/04cce012-ee1f-4910-a0f8-76d6e1d00118/volumes"
Oct 31 00:18:11.387153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.387066    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"]
Oct 31 00:18:11.387675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.387655    4478 cpu_manager.go:414] "containerMap: removing container" podUID="04cce012-ee1f-4910-a0f8-76d6e1d00118" containerName="maas-api"
Oct 31 00:18:11.387675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.387675    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04cce012-ee1f-4910-a0f8-76d6e1d00118" containerName="maas-api"
Oct 31 00:18:11.387871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.387777    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="04cce012-ee1f-4910-a0f8-76d6e1d00118" containerName="maas-api"
Oct 31 00:18:11.390968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.390945    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.402793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.402766    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.402898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.402807    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfxvn\" (UniqueName: \"kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.402952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.402890    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.402952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.402927    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.455409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.454635    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-10a7aa-serving-cert\""
Oct 31 00:18:11.462878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.462849    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"]
Oct 31 00:18:11.503911 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.503881    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.504052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.503915    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfxvn\" (UniqueName: \"kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.504052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.503940    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.504052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.503963    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.504151 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:11.504060    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-serving-cert: secret "maas-api-e2e-worker-w0-10a7aa-serving-cert" not found
Oct 31 00:18:11.504151 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:11.504142    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls podName:629634a5-2a44-4c5a-8f76-9e6c29c960c3 nodeName:}" failed. No retries permitted until 2026-10-31 00:18:12.004120995 +0000 UTC m=+925.020785685 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls") pod "maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" (UID: "629634a5-2a44-4c5a-8f76-9e6c29c960c3") : secret "maas-api-e2e-worker-w0-10a7aa-serving-cert" not found
Oct 31 00:18:11.504627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.504598    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.506602 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.506579    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:11.565524 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:11.565496    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfxvn\" (UniqueName: \"kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:12.009408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:12.009340    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:12.012687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:12.012614    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\") pod \"maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:12.303261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:12.303174    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:12.706176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:12.706067    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"]
Oct 31 00:18:12.709115 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:12.709089    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629634a5_2a44_4c5a_8f76_9e6c29c960c3.slice/crio-249e13a2878d5e475e380a037ee981c801364b506139cf9d2f9ffe2a7ecf984f WatchSource:0}: Error finding container 249e13a2878d5e475e380a037ee981c801364b506139cf9d2f9ffe2a7ecf984f: Status 404 returned error can't find the container with id 249e13a2878d5e475e380a037ee981c801364b506139cf9d2f9ffe2a7ecf984f
Oct 31 00:18:12.763005 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:12.762958    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" event={"ID":"629634a5-2a44-4c5a-8f76-9e6c29c960c3","Type":"ContainerStarted","Data":"249e13a2878d5e475e380a037ee981c801364b506139cf9d2f9ffe2a7ecf984f"}
Oct 31 00:18:13.730177 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:13.730151    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 00:18:13.769291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:13.769257    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" event={"ID":"629634a5-2a44-4c5a-8f76-9e6c29c960c3","Type":"ContainerStarted","Data":"33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6"}
Oct 31 00:18:13.769506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:13.769423    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:14.003160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:14.003096    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" podStartSLOduration=2.105403575 podStartE2EDuration="3.003062664s" totalImagesPullingTime="897.659089ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:14.002206879 +0000 UTC m=+927.018871610" watchObservedRunningTime="2026-10-31 00:18:14.003062664 +0000 UTC m=+927.019727377"
Oct 31 00:18:14.795792 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:14.795754    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:18:16.207034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.206998    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"]
Oct 31 00:18:16.207534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.207247    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg" podUID="b8b8988f-68ed-4b40-ab17-213440ed3cdd" containerName="maas-api" containerID="cri-o://c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e" gracePeriod=30
Oct 31 00:18:16.490531 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.490508    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:18:16.573841 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.573806    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\") pod \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") "
Oct 31 00:18:16.574033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.573864    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca\") pod \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") "
Oct 31 00:18:16.574033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.573915    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs\") pod \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") "
Oct 31 00:18:16.574033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.573996    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4\" (UniqueName: \"kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4\") pod \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\" (UID: \"b8b8988f-68ed-4b40-ab17-213440ed3cdd\") "
Oct 31 00:18:16.574308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.574278    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca" pod "b8b8988f-68ed-4b40-ab17-213440ed3cdd" (UID: "b8b8988f-68ed-4b40-ab17-213440ed3cdd"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:18:16.575099 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.575075    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b8988f-68ed-4b40-ab17-213440ed3cdd-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:16.576301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.576272    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls" pod "b8b8988f-68ed-4b40-ab17-213440ed3cdd" (UID: "b8b8988f-68ed-4b40-ab17-213440ed3cdd"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:16.576533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.576510    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs" pod "b8b8988f-68ed-4b40-ab17-213440ed3cdd" (UID: "b8b8988f-68ed-4b40-ab17-213440ed3cdd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:18:16.579494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.579466    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4" pod "b8b8988f-68ed-4b40-ab17-213440ed3cdd" (UID: "b8b8988f-68ed-4b40-ab17-213440ed3cdd"). InnerVolumeSpecName "kube-api-access-cnwt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:18:16.676920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.676881    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:16.676920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.676924    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8b8988f-68ed-4b40-ab17-213440ed3cdd-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:16.677133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.676941    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnwt4\" (UniqueName: \"kubernetes.io/projected/b8b8988f-68ed-4b40-ab17-213440ed3cdd-kube-api-access-cnwt4\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:18:16.791014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.790932    4478 generic.go:356] "Generic (PLEG): container finished" podID="b8b8988f-68ed-4b40-ab17-213440ed3cdd" containerID="c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e" exitCode=0
Oct 31 00:18:16.791014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.791006    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"
Oct 31 00:18:16.791238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.791010    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg" event={"ID":"b8b8988f-68ed-4b40-ab17-213440ed3cdd","Type":"ContainerDied","Data":"c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e"}
Oct 31 00:18:16.791238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.791064    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg" event={"ID":"b8b8988f-68ed-4b40-ab17-213440ed3cdd","Type":"ContainerDied","Data":"18c725bd00743f0e236d53139f20b826cd24f012186f65cc548fffae422ace22"}
Oct 31 00:18:16.791238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.791086    4478 scope.go:122] "RemoveContainer" containerID="c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e"
Oct 31 00:18:16.800573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.800550    4478 scope.go:122] "RemoveContainer" containerID="c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e"
Oct 31 00:18:16.800861 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:16.800837    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e\": container with ID starting with c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e not found: ID does not exist" containerID="c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e"
Oct 31 00:18:16.800939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.800870    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e"} err="failed to get container status \"c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e\": rpc error: code = NotFound desc = could not find container \"c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e\": container with ID starting with c376f7627b1c09bb18487717d01140ed839f09cd0e991243408e43900971c32e not found: ID does not exist"
Oct 31 00:18:16.955794 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.955760    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"]
Oct 31 00:18:16.959745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:16.959722    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b83a178e-d67749c6-svbjg"]
Oct 31 00:18:17.116987 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:17.116899    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b8988f-68ed-4b40-ab17-213440ed3cdd" path="/var/lib/kubelet/pods/b8b8988f-68ed-4b40-ab17-213440ed3cdd/volumes"
Oct 31 00:18:18.805115 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:18.805024    4478 generic.go:356] "Generic (PLEG): container finished" podID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerID="167c390049597f8c482fed6eace24d598b98e7b93d709bdf535566f612909871" exitCode=0
Oct 31 00:18:18.805588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:18.805113    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" event={"ID":"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8","Type":"ContainerDied","Data":"167c390049597f8c482fed6eace24d598b98e7b93d709bdf535566f612909871"}
Oct 31 00:18:18.805664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:18.805600    4478 scope.go:122] "RemoveContainer" containerID="167c390049597f8c482fed6eace24d598b98e7b93d709bdf535566f612909871"
Oct 31 00:18:18.806962 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:18.806932    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:18:18.807082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:18.807016    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:18:18.807082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:18.807065    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:18:19.784586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:19.784559    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 00:18:19.812150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:19.812118    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" event={"ID":"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8","Type":"ContainerStarted","Data":"1e4a10694110d5b11058bd0d046e195194769eef095efc362d8eda02d56ab3f7"}
Oct 31 00:18:19.814140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:19.814119    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:18:19.814403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:19.814306    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:18:19.814543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:19.814363    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:18:20.814077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:20.814049    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:18:20.814488 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:20.814101    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:18:21.814311 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:21.814274    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:18:21.814715 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:21.814337    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:18:22.764495 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.764458    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"]
Oct 31 00:18:22.765348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.765323    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b8b8988f-68ed-4b40-ab17-213440ed3cdd" containerName="maas-api"
Oct 31 00:18:22.780466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.780434    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b8988f-68ed-4b40-ab17-213440ed3cdd" containerName="maas-api"
Oct 31 00:18:22.780865 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.780840    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b8988f-68ed-4b40-ab17-213440ed3cdd" containerName="maas-api"
Oct 31 00:18:22.791420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.785008    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:22.813578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.813549    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:18:22.813679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.813612    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:18:22.819848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.819784    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"]
Oct 31 00:18:22.862995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.862958    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:22.863181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.863028    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z87bf\" (UniqueName: \"kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf\") pod \"payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:22.867153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.867108    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e1ca71\""
Oct 31 00:18:22.867520 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.867506    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e1ca71-dockercfg-54p24\""
Oct 31 00:18:22.964990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.955370    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"]
Oct 31 00:18:22.964990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.963738    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:22.964990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.964567    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:22.964990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.964587    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z87bf\" (UniqueName: \"kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf\") pod \"payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:22.968745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.968723    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:22.999314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:22.999283    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"]
Oct 31 00:18:23.025427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.025347    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z87bf\" (UniqueName: \"kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf\") pod \"payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:23.066004 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.065974    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.066159 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.066019    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnl9m\" (UniqueName: \"kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m\") pod \"payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.101676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.101650    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:23.168829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.168786    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.169013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.168842    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnl9m\" (UniqueName: \"kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m\") pod \"payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.169617 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.169595    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.229902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.229868    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnl9m\" (UniqueName: \"kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m\") pod \"payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.282867 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.282779    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.512348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.512317    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"]
Oct 31 00:18:23.515072 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:23.515037    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8bbd7ff_2c42_4c29_94ad_ba7462e0b563.slice/crio-7c74991618f0247daf723c1918d72c9a042f0a197f1c1aab7302e34b259416c6 WatchSource:0}: Error finding container 7c74991618f0247daf723c1918d72c9a042f0a197f1c1aab7302e34b259416c6: Status 404 returned error can't find the container with id 7c74991618f0247daf723c1918d72c9a042f0a197f1c1aab7302e34b259416c6
Oct 31 00:18:23.683587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.683554    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"]
Oct 31 00:18:23.689329 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:23.689298    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb62e589_ff36_4754_afe9_c0e7cc9dbf0e.slice/crio-3a0ad78dc659f215141a309f9ec0a9a1caba2437e98157f389b8ce82bc4d4b0f WatchSource:0}: Error finding container 3a0ad78dc659f215141a309f9ec0a9a1caba2437e98157f389b8ce82bc4d4b0f: Status 404 returned error can't find the container with id 3a0ad78dc659f215141a309f9ec0a9a1caba2437e98157f389b8ce82bc4d4b0f
Oct 31 00:18:23.813706 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.813674    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:18:23.813842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.813721    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:18:23.849451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.844822    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" event={"ID":"db62e589-ff36-4754-afe9-c0e7cc9dbf0e","Type":"ContainerStarted","Data":"fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805"}
Oct 31 00:18:23.849451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.844888    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" event={"ID":"db62e589-ff36-4754-afe9-c0e7cc9dbf0e","Type":"ContainerStarted","Data":"3a0ad78dc659f215141a309f9ec0a9a1caba2437e98157f389b8ce82bc4d4b0f"}
Oct 31 00:18:23.849451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.845785    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:23.861408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.852660    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" event={"ID":"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563","Type":"ContainerStarted","Data":"a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29"}
Oct 31 00:18:23.861408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.852716    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" event={"ID":"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563","Type":"ContainerStarted","Data":"7c74991618f0247daf723c1918d72c9a042f0a197f1c1aab7302e34b259416c6"}
Oct 31 00:18:23.861408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.853413    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:23.964306 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:23.964244    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" podStartSLOduration=1.9642238220000001 podStartE2EDuration="1.964223822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:23.961649746 +0000 UTC m=+936.978314483" watchObservedRunningTime="2026-10-31 00:18:23.964223822 +0000 UTC m=+936.980888535"
Oct 31 00:18:24.086999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:24.086573    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" podStartSLOduration=2.086535411 podStartE2EDuration="2.086535411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:24.085528964 +0000 UTC m=+937.102193695" watchObservedRunningTime="2026-10-31 00:18:24.086535411 +0000 UTC m=+937.103200125"
Oct 31 00:18:24.813666 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:24.813633    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Oct 31 00:18:24.813852 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:24.813686    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Oct 31 00:18:25.818028 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:25.818003    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:18:25.818459 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:25.818411    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:18:25.819041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:25.819022    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 00:18:35.866443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:35.866351    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 00:18:35.866924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:35.866456    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 00:18:39.186186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.186142    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"]
Oct 31 00:18:39.189675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.189652    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.241351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.241306    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-e1ca71-serving-cert\""
Oct 31 00:18:39.249612 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.249577    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.249759 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.249644    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.249808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.249760    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cms45\" (UniqueName: \"kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.249844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.249808    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.294067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.294030    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"]
Oct 31 00:18:39.352824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.352787    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.353007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.352844    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.353007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.352893    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cms45\" (UniqueName: \"kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.353007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.352923    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.353007 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:39.352958    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-serving-cert: secret "maas-api-e2e-worker-w2-e1ca71-serving-cert" not found
Oct 31 00:18:39.353241 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:39.353026    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls podName:3d3cbbb4-30fb-451d-b2b8-a232b26de18c nodeName:}" failed. No retries permitted until 2026-10-31 00:18:39.85300426 +0000 UTC m=+952.869668972 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls") pod "maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" (UID: "3d3cbbb4-30fb-451d-b2b8-a232b26de18c") : secret "maas-api-e2e-worker-w2-e1ca71-serving-cert" not found
Oct 31 00:18:39.353725 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.353702    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.356358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.356328    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.419325 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.419291    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cms45\" (UniqueName: \"kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.863447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.863406    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:39.878833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:39.878788    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:40.103585 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:40.103522    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:40.480336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:40.480303    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"]
Oct 31 00:18:40.483489 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:40.483461    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d3cbbb4_30fb_451d_b2b8_a232b26de18c.slice/crio-c5790159438df75bc4c9c9f21e443f7d68bc958e2b6099af675f58ba029cce21 WatchSource:0}: Error finding container c5790159438df75bc4c9c9f21e443f7d68bc958e2b6099af675f58ba029cce21: Status 404 returned error can't find the container with id c5790159438df75bc4c9c9f21e443f7d68bc958e2b6099af675f58ba029cce21
Oct 31 00:18:40.945101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:40.945064    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" event={"ID":"3d3cbbb4-30fb-451d-b2b8-a232b26de18c","Type":"ContainerStarted","Data":"c5790159438df75bc4c9c9f21e443f7d68bc958e2b6099af675f58ba029cce21"}
Oct 31 00:18:41.951635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:41.951598    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" event={"ID":"3d3cbbb4-30fb-451d-b2b8-a232b26de18c","Type":"ContainerStarted","Data":"8380ea0a4624efc9495e4c41e7288d04b9923bbe75b14f62590810c487e5906a"}
Oct 31 00:18:41.952015 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:41.951673    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:42.070788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:42.070740    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" podStartSLOduration=2.148364379 podStartE2EDuration="3.070721829s" totalImagesPullingTime="922.35745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:42.069892977 +0000 UTC m=+955.086557808" watchObservedRunningTime="2026-10-31 00:18:42.070721829 +0000 UTC m=+955.087386541"
Oct 31 00:18:44.756678 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:44.756646    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:18:47.962427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:47.962399    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 00:18:49.379639 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.379601    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"]
Oct 31 00:18:49.384718 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.384693    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.457207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.457179    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.457370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.457232    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.457370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.457329    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsj4p\" (UniqueName: \"kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.457511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.457444    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.476545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.476512    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-869cca-serving-cert\""
Oct 31 00:18:49.478731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.478705    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"]
Oct 31 00:18:49.558757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.558723    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.558945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.558817    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.558945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.558867    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.558945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.558905    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsj4p\" (UniqueName: \"kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.559110 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:49.559048    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-serving-cert: secret "maas-api-e2e-worker-w1-869cca-serving-cert" not found
Oct 31 00:18:49.559159 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:18:49.559126    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls podName:f5fa99f3-efc9-4179-9413-de74dbf122df nodeName:}" failed. No retries permitted until 2026-10-31 00:18:50.059105207 +0000 UTC m=+963.075769914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls") pod "maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" (UID: "f5fa99f3-efc9-4179-9413-de74dbf122df") : secret "maas-api-e2e-worker-w1-869cca-serving-cert" not found
Oct 31 00:18:49.559513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.559493    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.561415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.561356    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:49.620550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:49.620523    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsj4p\" (UniqueName: \"kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:50.063702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:50.063660    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:50.066143 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:50.066120    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\") pod \"maas-api-e2e-worker-w1-869cca-6974644485-7gjnz\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:50.298083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:50.298053    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:50.673042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:50.672966    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"]
Oct 31 00:18:50.675905 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:18:50.675880    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fa99f3_efc9_4179_9413_de74dbf122df.slice/crio-0876e88fbe17bbf8e45b858918fcd2e3c739296e4157831e1e481c504ad43ef2 WatchSource:0}: Error finding container 0876e88fbe17bbf8e45b858918fcd2e3c739296e4157831e1e481c504ad43ef2: Status 404 returned error can't find the container with id 0876e88fbe17bbf8e45b858918fcd2e3c739296e4157831e1e481c504ad43ef2
Oct 31 00:18:50.995182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:50.995147    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" event={"ID":"f5fa99f3-efc9-4179-9413-de74dbf122df","Type":"ContainerStarted","Data":"0876e88fbe17bbf8e45b858918fcd2e3c739296e4157831e1e481c504ad43ef2"}
Oct 31 00:18:52.000825 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:52.000795    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" event={"ID":"f5fa99f3-efc9-4179-9413-de74dbf122df","Type":"ContainerStarted","Data":"77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080"}
Oct 31 00:18:52.001267 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:52.000900    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:18:52.120059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:52.120002    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" podStartSLOduration=2.281093838 podStartE2EDuration="3.119988794s" totalImagesPullingTime="838.894956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:52.117672322 +0000 UTC m=+965.134337033" watchObservedRunningTime="2026-10-31 00:18:52.119988794 +0000 UTC m=+965.136653506"
Oct 31 00:18:58.010944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:18:58.010911    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 00:19:14.756231 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:19:14.756186    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:19:44.756845 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:19:44.756809    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:20:14.759650 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:20:14.759557    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:20:44.759497 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:20:44.759454    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:21:14.027502 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:14.027466    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"]
Oct 31 00:21:14.028078 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:14.027695    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy" containerID="cri-o://3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82" gracePeriod=30
Oct 31 00:21:14.759321 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:21:14.759285    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:21:19.299043 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.298985    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346235    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346295    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346354    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346424    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346463    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346506    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346542    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346616    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.346652    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy\") pod \"0cf5bcbc-4566-490d-a288-f1661106fb7b\" (UID: \"0cf5bcbc-4566-490d-a288-f1661106fb7b\") "
Oct 31 00:21:19.353030 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.352650    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:19.355171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.355135    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "kube-api-access-s7tdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:19.355528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.355503    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:21:19.355782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.355761    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:19.355967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.355948    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:19.356151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.356132    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:19.364414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.362628    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:21:19.374163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.372689    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:21:19.374854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.374830    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy" pod "0cf5bcbc-4566-490d-a288-f1661106fb7b" (UID: "0cf5bcbc-4566-490d-a288-f1661106fb7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:21:19.447898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447870    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cf5bcbc-4566-490d-a288-f1661106fb7b-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.447898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447895    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.447898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447905    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.448109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447915    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.448109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447925    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.448109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447934    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.448109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447942    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.448109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447951    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cf5bcbc-4566-490d-a288-f1661106fb7b-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.448109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.447959    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7tdr\" (UniqueName: \"kubernetes.io/projected/0cf5bcbc-4566-490d-a288-f1661106fb7b-kube-api-access-s7tdr\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:21:19.731571 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.731534    4478 generic.go:356] "Generic (PLEG): container finished" podID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerID="3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82" exitCode=0
Oct 31 00:21:19.731855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.731599    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"
Oct 31 00:21:19.731855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.731624    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" event={"ID":"0cf5bcbc-4566-490d-a288-f1661106fb7b","Type":"ContainerDied","Data":"3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82"}
Oct 31 00:21:19.731855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.731719    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r" event={"ID":"0cf5bcbc-4566-490d-a288-f1661106fb7b","Type":"ContainerDied","Data":"c4dcb4cd7f1ed112bff0a86f2dcc5afb73ff118320e1af2143f787ba10042043"}
Oct 31 00:21:19.731855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.731752    4478 scope.go:122] "RemoveContainer" containerID="3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82"
Oct 31 00:21:19.741460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.741445    4478 scope.go:122] "RemoveContainer" containerID="3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82"
Oct 31 00:21:19.741751 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:21:19.741735    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82\": container with ID starting with 3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82 not found: ID does not exist" containerID="3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82"
Oct 31 00:21:19.741805 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.741759    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82"} err="failed to get container status \"3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82\": rpc error: code = NotFound desc = could not find container \"3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82\": container with ID starting with 3f94672551069a6e8deb7df780622497fb3f1903c9c454e48180aebc39fa0f82 not found: ID does not exist"
Oct 31 00:21:19.896278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.896249    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"]
Oct 31 00:21:19.945529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:19.945498    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4kf6r"]
Oct 31 00:21:21.116878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:21.116846    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" path="/var/lib/kubelet/pods/0cf5bcbc-4566-490d-a288-f1661106fb7b/volumes"
Oct 31 00:21:44.761489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:21:44.761453    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:21:54.558506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.558460    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"]
Oct 31 00:21:54.559073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.558921    4478 cpu_manager.go:414] "containerMap: removing container" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy"
Oct 31 00:21:54.559073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.558933    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy"
Oct 31 00:21:54.559073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.559022    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cf5bcbc-4566-490d-a288-f1661106fb7b" containerName="istio-proxy"
Oct 31 00:21:54.563257 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.563233    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.615547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.615500    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a2f1ffbe7a37\""
Oct 31 00:21:54.665125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.665091    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 00:21:54.665301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.665135    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 00:21:54.665301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.665152    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h8pxf\""
Oct 31 00:21:54.665301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.665134    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-aa8816631626\""
Oct 31 00:21:54.665301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.665173    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 00:21:54.665301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.665211    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 00:21:54.667710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.667688    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"]
Oct 31 00:21:54.699538 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.699504    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.699734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.699612    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.699734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.699711    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.699879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.699772    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.801088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.801053    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.801288 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.801107    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.801288 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.801184    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.801288 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.801220    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.801968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.801865    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.802158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.802113    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.803843 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.803819    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.868101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.868023    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg\") pod \"payload-processing-external-model-7cd8774477-vdcld\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") " pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:54.878882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:54.878855    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:55.274916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:55.274857    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"]
Oct 31 00:21:55.277612 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:21:55.277582    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251ce9dd_a54e_474f_b841_c54556186e0a.slice/crio-f8027965126600d3ff237872eb40974725e7d1231c35ce75fb84316f9f7b7c6f WatchSource:0}: Error finding container f8027965126600d3ff237872eb40974725e7d1231c35ce75fb84316f9f7b7c6f: Status 404 returned error can't find the container with id f8027965126600d3ff237872eb40974725e7d1231c35ce75fb84316f9f7b7c6f
Oct 31 00:21:55.930831 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:55.930783    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld" event={"ID":"251ce9dd-a54e-474f-b841-c54556186e0a","Type":"ContainerStarted","Data":"5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f"}
Oct 31 00:21:55.931334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:55.930857    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld" event={"ID":"251ce9dd-a54e-474f-b841-c54556186e0a","Type":"ContainerStarted","Data":"f8027965126600d3ff237872eb40974725e7d1231c35ce75fb84316f9f7b7c6f"}
Oct 31 00:21:55.931334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:55.930902    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:21:56.048554 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:21:56.048053    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld" podStartSLOduration=2.048035487 podStartE2EDuration="2.048035487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:56.047441698 +0000 UTC m=+1149.064106411" watchObservedRunningTime="2026-10-31 00:21:56.048035487 +0000 UTC m=+1149.064700212"
Oct 31 00:22:00.429432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.429403    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"]
Oct 31 00:22:00.433335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.433310    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.485361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.485331    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-10a7aa-openshift-default-dockercfg-zqppq\""
Oct 31 00:22:00.537780 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.537747    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"]
Oct 31 00:22:00.559875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.559839    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560016 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.559961    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560017    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560065    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560202    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560293 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560254    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560370    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxn5j\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560445    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.560488 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.560466    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661711 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661668    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661711 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661745    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661778    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661797    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661839    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxn5j\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661877    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661901    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.661941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.661938    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.662317 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.662199    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.662628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.662448    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.662628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.662476    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.662628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.662574    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.663036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.663012    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.664350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.664325    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.664694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.664670    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.724933 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.724870    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.725116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.725094    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxn5j\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:00.748788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:00.748766    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:01.177855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:01.177818    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"]
Oct 31 00:22:01.180967 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:22:01.180940    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda1c2756_594d_4018_a886_260b76006523.slice/crio-47ef7f5da91dca9ecb86f8dd61f98a35ebc48417f17d005a2cfa0241970c7040 WatchSource:0}: Error finding container 47ef7f5da91dca9ecb86f8dd61f98a35ebc48417f17d005a2cfa0241970c7040: Status 404 returned error can't find the container with id 47ef7f5da91dca9ecb86f8dd61f98a35ebc48417f17d005a2cfa0241970c7040
Oct 31 00:22:01.183257 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:01.183227    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:01.183329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:01.183290    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:01.183329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:01.183322    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:01.961726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:01.961688    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" event={"ID":"da1c2756-594d-4018-a886-260b76006523","Type":"ContainerStarted","Data":"1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211"}
Oct 31 00:22:01.962132 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:01.961752    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" event={"ID":"da1c2756-594d-4018-a886-260b76006523","Type":"ContainerStarted","Data":"47ef7f5da91dca9ecb86f8dd61f98a35ebc48417f17d005a2cfa0241970c7040"}
Oct 31 00:22:02.180672 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:02.180605    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podStartSLOduration=2.180592463 podStartE2EDuration="2.180592463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:02.13060344 +0000 UTC m=+1155.147268214" watchObservedRunningTime="2026-10-31 00:22:02.180592463 +0000 UTC m=+1155.197257174"
Oct 31 00:22:02.754031 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:02.749779    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:02.756541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:02.756502    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:22:02.756656 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:02.756592    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:22:03.753408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:03.751801    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:22:03.753408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:03.751871    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:22:04.753409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:04.750784    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:22:04.753409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:04.750876    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:22:05.753407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:05.750816    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:22:05.753407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:05.750898    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:22:06.755409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:06.751766    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body=
Oct 31 00:22:06.755409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:06.751830    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused"
Oct 31 00:22:06.957186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:06.957154    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:22:07.860501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:07.860472    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:07.861513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:07.861486    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:07.862519 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:07.862494    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:22:14.500533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.500500    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"]
Oct 31 00:22:14.503998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.503976    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.550891 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.550854    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"]
Oct 31 00:22:14.555133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.555113    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.600997 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.600945    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601008    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601039    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601111    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601145    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601240    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601394    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601435    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601538    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhgvd\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601594    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601727 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601708    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601756    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nssdh\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601779    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601818    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.601922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601847    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.601922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601886    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.602013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601926    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.602013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.601958    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.610739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.610706    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"]
Oct 31 00:22:14.661910 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.661878    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"]
Oct 31 00:22:14.706552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706516    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.706748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706572    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.706748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706605    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.706748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706684    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.706748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.706748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706738    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706770    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706802    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706857    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706899    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706938    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.706974    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhgvd\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707024    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707107    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707172    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nssdh\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.707432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707202    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707238    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.707432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707265    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.707787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.707763    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.709374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.708346    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.709374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.708579    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.709374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.708635    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.709374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.709245    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.711959 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.710320    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.716573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.711101    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.716724 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.711112    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.717113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.711282    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.717248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.711623    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.717332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.711709    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.717675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.717645    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.717839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.717817    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.718025 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.717998    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.780528 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:22:14.780054    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:22:14.782641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.782610    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.789740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.786474    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.789740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.788808    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhgvd\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd\") pod \"maas-default-gateway-openshift-default-58c6585447-4znwv\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:14.789740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.789647    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nssdh\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh\") pod \"maas-default-gateway-openshift-default-58c6585447-j6ldv\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.817190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.817171    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:14.872838 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:14.872805    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:15.264839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.264804    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"]
Oct 31 00:22:15.306672 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:22:15.306643    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3857a068_f40e_47be_a7a8_7c9e71c46daf.slice/crio-06fc5a016bbff112a1436e3bceeac78c664fe76abb2f3888968ba2140f3cab69 WatchSource:0}: Error finding container 06fc5a016bbff112a1436e3bceeac78c664fe76abb2f3888968ba2140f3cab69: Status 404 returned error can't find the container with id 06fc5a016bbff112a1436e3bceeac78c664fe76abb2f3888968ba2140f3cab69
Oct 31 00:22:15.309283 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.309248    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:15.309369 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.309324    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:15.309369 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.309354    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:15.322076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.321625    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"]
Oct 31 00:22:15.324181 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:22:15.324159    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3f7644_c2cd_43fa_81f6_ec257f7eb5b8.slice/crio-cec76eaadfe11a0700823a6ad14bac82545155d31188ea3330d0bbc3a353047f WatchSource:0}: Error finding container cec76eaadfe11a0700823a6ad14bac82545155d31188ea3330d0bbc3a353047f: Status 404 returned error can't find the container with id cec76eaadfe11a0700823a6ad14bac82545155d31188ea3330d0bbc3a353047f
Oct 31 00:22:15.327122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.327089    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:15.327219 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.327172    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:15.327285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:15.327217    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:16.031720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.031677    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" event={"ID":"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8","Type":"ContainerStarted","Data":"8072c9a02684e826ecf2fb4815753d628c898ed0cbe334e7c59bb74d6bfc0aa4"}
Oct 31 00:22:16.032217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.031749    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" event={"ID":"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8","Type":"ContainerStarted","Data":"cec76eaadfe11a0700823a6ad14bac82545155d31188ea3330d0bbc3a353047f"}
Oct 31 00:22:16.045406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.044853    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" event={"ID":"3857a068-f40e-47be-a7a8-7c9e71c46daf","Type":"ContainerStarted","Data":"e31859904895a8a7f295742ae59092824f3cfeb71800e4e9ef38c82219be376a"}
Oct 31 00:22:16.045406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.044923    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" event={"ID":"3857a068-f40e-47be-a7a8-7c9e71c46daf","Type":"ContainerStarted","Data":"06fc5a016bbff112a1436e3bceeac78c664fe76abb2f3888968ba2140f3cab69"}
Oct 31 00:22:16.162406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.162248    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podStartSLOduration=2.162225859 podStartE2EDuration="2.162225859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:16.159716658 +0000 UTC m=+1169.176381373" watchObservedRunningTime="2026-10-31 00:22:16.162225859 +0000 UTC m=+1169.178890572"
Oct 31 00:22:16.284462 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.281675    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podStartSLOduration=2.281653674 podStartE2EDuration="2.281653674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:16.280759133 +0000 UTC m=+1169.297423869" watchObservedRunningTime="2026-10-31 00:22:16.281653674 +0000 UTC m=+1169.298318387"
Oct 31 00:22:16.818411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.818057    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:16.820112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.820070    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:16.820235 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.820135    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:16.878403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.877258    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:16.878403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.877406    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:16.878403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:16.877453    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:17.823078 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:17.822540    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:17.823078 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:17.822604    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:17.886457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:17.885324    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:17.886457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:17.885412    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:18.827407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:18.818174    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:18.827407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:18.818259    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:18.891410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:18.879887    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:18.891410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:18.880035    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:19.817874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:19.817828    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:19.818116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:19.818097    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:19.877408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:19.876783    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:19.877408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:19.876858    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:20.817569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:20.817535    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:20.817782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:20.817591    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:20.873797 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:20.873763    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:20.874104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:20.874058    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:21.818025 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:21.817986    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:21.818564 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:21.818051    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:21.873357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:21.873322    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:21.873575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:21.873407    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:22.817857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:22.817818    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Oct 31 00:22:22.818039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:22.817887    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Oct 31 00:22:22.874424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:22.874338    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Oct 31 00:22:22.874849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:22.874466    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Oct 31 00:22:24.057837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:24.057812    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:24.059607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:24.059582    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:24.060828 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:24.060793    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:22:24.188489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:24.188454    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:24.189040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:24.188952    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:24.191399 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:24.191155    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:22:29.674190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.674136    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"]
Oct 31 00:22:29.680371 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.680348    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.733710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.733676    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.733852 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.733753    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.733894 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.733861    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.733939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.733922    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.733978 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.733955    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w22mp\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.734011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.733998    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.734044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.734023    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.734077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.734048    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.734109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.734096    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.807115 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.806585    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"]
Oct 31 00:22:29.835510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835475    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.835705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835533    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.835705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835578    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.835705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835629    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w22mp\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.835705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835668    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.835705 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835696    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.836020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835719    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.836020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835766    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.836020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.835898    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.836418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.836361    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.837252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.836679    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.837252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.836854    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.837252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.837208    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.837904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.837886    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.844726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.839932    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.844726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.844656    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.896988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.896957    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:29.897182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:29.897160    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w22mp\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp\") pod \"maas-default-gateway-openshift-default-58c6585447-cpsx2\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:30.000313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.000232    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:30.433857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.433828    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"]
Oct 31 00:22:30.440134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.440101    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:30.440250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.440183    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:30.440250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.440228    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:30.676627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.676591    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"]
Oct 31 00:22:30.680839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.680822    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.745580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.745541    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e1ca71-openshift-default-dockercfg-sh7lb\""
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.749782    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.749845    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.749881    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.749915    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.749978    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.750017    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.750056    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.750142    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pbxc\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.756119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.750185    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.797687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.797655    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"]
Oct 31 00:22:30.852086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.851821    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.851876    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.851920    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852001    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pbxc\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852041    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852092    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852133    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852160    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852190    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.852723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852632    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.853181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.853110    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.853330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.853309    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.853813 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.853790    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.853931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.852868    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.855818 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.855796    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.856140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.856116    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.914772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.914739    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.914913 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.914884    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pbxc\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:30.996807 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:30.996727    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:31.196740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.196691    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" event={"ID":"19ddd893-2592-4ec8-b8e8-8b4030d831a8","Type":"ContainerStarted","Data":"13eb0e55154f6a3598ee594d4c8c6fe4594f9c45772c315e0168e5d5ebc76737"}
Oct 31 00:22:31.196901 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.196759    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" event={"ID":"19ddd893-2592-4ec8-b8e8-8b4030d831a8","Type":"ContainerStarted","Data":"802d21401c80066aa6fd979eb07cdbfd6d818b5f8cc2a3a315780bab6f85ea02"}
Oct 31 00:22:31.328599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.328505    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podStartSLOduration=2.328490043 podStartE2EDuration="2.328490043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:31.327578885 +0000 UTC m=+1184.344243645" watchObservedRunningTime="2026-10-31 00:22:31.328490043 +0000 UTC m=+1184.345154815"
Oct 31 00:22:31.474138 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.474077    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"]
Oct 31 00:22:31.476514 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:22:31.476487    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode397d119_10c6_4fef_960b_aea1b23923cd.slice/crio-6330f8da4019010d398dab1085e1e6e01260715a2182e6a036186060b0b6034d WatchSource:0}: Error finding container 6330f8da4019010d398dab1085e1e6e01260715a2182e6a036186060b0b6034d: Status 404 returned error can't find the container with id 6330f8da4019010d398dab1085e1e6e01260715a2182e6a036186060b0b6034d
Oct 31 00:22:31.478868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.478827    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:31.478952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.478903    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:31.478952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:31.478931    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:22:32.000870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.000838    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:32.002256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.002231    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:32.002449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.002282    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:32.204329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.204292    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" event={"ID":"e397d119-10c6-4fef-960b-aea1b23923cd","Type":"ContainerStarted","Data":"3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92"}
Oct 31 00:22:32.204528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.204350    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" event={"ID":"e397d119-10c6-4fef-960b-aea1b23923cd","Type":"ContainerStarted","Data":"6330f8da4019010d398dab1085e1e6e01260715a2182e6a036186060b0b6034d"}
Oct 31 00:22:32.325053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.324954    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podStartSLOduration=2.324936574 podStartE2EDuration="2.324936574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:32.324204109 +0000 UTC m=+1185.340868836" watchObservedRunningTime="2026-10-31 00:22:32.324936574 +0000 UTC m=+1185.341601287"
Oct 31 00:22:32.997081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.996993    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:32.998782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.998762    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:22:32.998868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:32.998819    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 00:22:33.001085 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:33.001062    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:33.001447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:33.001202    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:33.998027 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:33.997713    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:22:33.998027 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:33.997757    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 00:22:34.001422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:34.001329    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:34.001422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:34.001364    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:34.997213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:34.997177    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:22:34.997437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:34.997227    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 00:22:35.001338 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:35.001319    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:35.001428 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:35.001352    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:35.997939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:35.997898    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:22:35.998137 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:35.997954    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 00:22:36.000640 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:36.000617    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:36.000691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:36.000664    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:36.998094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:36.998050    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:22:36.998594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:36.998118    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 00:22:37.001394 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:37.001349    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:37.001540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:37.001407    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:37.997011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:37.996977    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 31 00:22:37.997165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:37.997031    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 31 00:22:38.001375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:38.001352    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body=
Oct 31 00:22:38.001691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:38.001426    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused"
Oct 31 00:22:39.003555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:39.003521    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:39.003976 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:39.003910    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:39.004738 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:39.004714    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:22:39.057370 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:39.057344    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:39.057754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:39.057731    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:39.058815 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:22:39.058796    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:22:44.758753 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:22:44.758711    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:23:14.150571 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.150530    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a2f1ffbe7a37: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:14.150571 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.150567    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:14.151285 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.150657    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:23:14.650637129 +0000 UTC m=+1227.667301832 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:14.655772 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.655741    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a2f1ffbe7a37: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:14.655772 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.655773    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:14.656035 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.655837    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:23:15.655820691 +0000 UTC m=+1228.672485394 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:14.758866 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:14.758820    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:23:15.669516 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:15.669484    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a2f1ffbe7a37: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:15.670119 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:15.670098    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:15.670319 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:15.670308    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:23:17.670269957 +0000 UTC m=+1230.686934661 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:17.694928 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:17.694891    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-a2f1ffbe7a37: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:17.694928 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:17.694919    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:17.695416 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:17.694998    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:23:21.694984031 +0000 UTC m=+1234.711648722 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-a2f1ffbe7a37" not found
Oct 31 00:23:21.746463 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:21.746430    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-aa8816631626: secret "agc-cred-aa8816631626" not found
Oct 31 00:23:21.746463 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:21.746454    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-aa8816631626" not found
Oct 31 00:23:21.748906 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:21.746506    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:23:29.746492374 +0000 UTC m=+1242.763157078 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-aa8816631626" not found
Oct 31 00:23:29.829544 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:29.829514    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-aa8816631626: secret "agc-cred-aa8816631626" not found
Oct 31 00:23:29.829544 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:29.829538    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-aa8816631626" not found
Oct 31 00:23:29.829975 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:29.829606    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:23:45.829592519 +0000 UTC m=+1258.846257208 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-aa8816631626" not found
Oct 31 00:23:37.727848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.727813    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"]
Oct 31 00:23:37.731399 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.731366    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767371 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767342    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767413    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767471    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767536    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767615    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767682 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767650    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lspjq\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767680    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767713    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.767826 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.767739    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.783642 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.783620    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-869cca-openshift-default-dockercfg-49wg8\""
Oct 31 00:23:37.836003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.835977    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"]
Oct 31 00:23:37.868945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.868921    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.868962    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869149 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869089    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869149 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869135    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869254 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869215    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869337    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869605 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869574    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lspjq\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869626    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869643    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869664    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.869776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869706    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.870138 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.869937    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.870138 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.870002    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.870551 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.870529    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.871812 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.871794    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.871910 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.871846    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.951776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.950092    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lspjq\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:37.953533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:37.953469    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:38.044731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.044658    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:38.492543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.492461    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"]
Oct 31 00:23:38.494769 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:23:38.494742    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4037f26_6727_4523_b6b5_6c7af9a5f771.slice/crio-ed1f7591de7fe5cc69a55eb1457510370d58f3142cf1edece94df2df1f7b9aa4 WatchSource:0}: Error finding container ed1f7591de7fe5cc69a55eb1457510370d58f3142cf1edece94df2df1f7b9aa4: Status 404 returned error can't find the container with id ed1f7591de7fe5cc69a55eb1457510370d58f3142cf1edece94df2df1f7b9aa4
Oct 31 00:23:38.497150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.497120    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:23:38.497225 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.497185    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:23:38.497225 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.497212    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:23:38.626730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.626695    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" event={"ID":"d4037f26-6727-4523-b6b5-6c7af9a5f771","Type":"ContainerStarted","Data":"dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff"}
Oct 31 00:23:38.626906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.626761    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" event={"ID":"d4037f26-6727-4523-b6b5-6c7af9a5f771","Type":"ContainerStarted","Data":"ed1f7591de7fe5cc69a55eb1457510370d58f3142cf1edece94df2df1f7b9aa4"}
Oct 31 00:23:38.748034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:38.747892    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podStartSLOduration=1.7478755719999999 podStartE2EDuration="1.747875572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:38.747210536 +0000 UTC m=+1251.763875271" watchObservedRunningTime="2026-10-31 00:23:38.747875572 +0000 UTC m=+1251.764540283"
Oct 31 00:23:39.046851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:39.046749    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:39.051535 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:39.051501    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:39.051822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:39.051795    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:40.045789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:40.045752    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:40.046403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:40.046350    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:41.051405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:41.050791    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:41.051405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:41.050850    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:42.047407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:42.046867    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:42.047407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:42.046935    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:43.048787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:43.048750    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:43.049400 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:43.049344    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:44.055275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:44.053736    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:44.055897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:44.055864    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:44.771244 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:44.771170    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:23:45.046440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:45.045876    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 31 00:23:45.046440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:45.045941    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 31 00:23:45.902957 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:45.902521    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-aa8816631626: secret "agc-cred-aa8816631626" not found
Oct 31 00:23:45.902957 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:45.902553    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-aa8816631626" not found
Oct 31 00:23:45.902957 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:23:45.902614    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:24:17.902595202 +0000 UTC m=+1290.919259906 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-aa8816631626" not found
Oct 31 00:23:46.064697 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:46.064663    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:46.066211 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:46.066188    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:23:46.073158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:23:46.073139    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 00:24:01.443213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.443180    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"]
Oct 31 00:24:01.448026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.447998    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.560199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.560169    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"]
Oct 31 00:24:01.583762 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.583729    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.583927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.583806    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.583927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.583899    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.584012 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.583997    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.584053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.584038    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.584110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.584094    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.584150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.584140    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.584189 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.584156    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.584222 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.584191    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq8hk\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685218 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685188    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685233    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685248    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685268    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq8hk\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685304    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685363    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685419    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685480    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685523    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685663    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685819    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.685999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.685935    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.686057 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.686028    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.686157 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.686138    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.687791 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.687767    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.687916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.687885    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.746820 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.746745    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.746942 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.746866    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq8hk\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:01.761582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:01.761559    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:02.194209 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.194177    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"]
Oct 31 00:24:02.197867 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:24:02.197840    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb1d2f0_7401_47bb_b522_614f0486f30c.slice/crio-284e0a4b7c716e369130e3ee78f0d77d4d95cdfdbc986aa45e5ef27197e2be82 WatchSource:0}: Error finding container 284e0a4b7c716e369130e3ee78f0d77d4d95cdfdbc986aa45e5ef27197e2be82: Status 404 returned error can't find the container with id 284e0a4b7c716e369130e3ee78f0d77d4d95cdfdbc986aa45e5ef27197e2be82
Oct 31 00:24:02.200164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.200131    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:24:02.200218 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.200196    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:24:02.200264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.200226    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:24:02.791683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.791648    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" event={"ID":"9cb1d2f0-7401-47bb-b522-614f0486f30c","Type":"ContainerStarted","Data":"a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca"}
Oct 31 00:24:02.792067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.791701    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" event={"ID":"9cb1d2f0-7401-47bb-b522-614f0486f30c","Type":"ContainerStarted","Data":"284e0a4b7c716e369130e3ee78f0d77d4d95cdfdbc986aa45e5ef27197e2be82"}
Oct 31 00:24:02.912989 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:02.912935    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podStartSLOduration=1.912920204 podStartE2EDuration="1.912920204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:02.912097571 +0000 UTC m=+1275.928762287" watchObservedRunningTime="2026-10-31 00:24:02.912920204 +0000 UTC m=+1275.929584979"
Oct 31 00:24:03.762579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:03.762542    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:03.764300 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:03.764270    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:24:03.764474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:03.764326    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 31 00:24:04.762879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:04.762841    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:24:04.763366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:04.762902    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 31 00:24:05.762829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:05.762791    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:24:05.763052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:05.762866    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 31 00:24:06.762950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:06.762918    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:24:06.763158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:06.762991    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 31 00:24:07.763434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:07.762781    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:24:07.763434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:07.762846    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 31 00:24:08.762080 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:08.762044    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 31 00:24:08.762240 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:08.762111    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 31 00:24:09.766540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:09.766511    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:09.766955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:09.766909    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:09.767585 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:09.767568    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:24:14.763208 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:24:14.763170    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:24:17.910022 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:24:17.909991    4478 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-aa8816631626: secret "agc-cred-aa8816631626" not found
Oct 31 00:24:17.910022 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:24:17.910013    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-7cd8774477-vdcld: secret "agc-cred-aa8816631626" not found
Oct 31 00:24:17.910545 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:24:17.910061    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials podName:251ce9dd-a54e-474f-b841-c54556186e0a nodeName:}" failed. No retries permitted until 2026-10-31 00:25:21.910047478 +0000 UTC m=+1354.926712169 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials") pod "payload-processing-external-model-7cd8774477-vdcld" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a") : secret "agc-cred-aa8816631626" not found
Oct 31 00:24:40.404947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.404904    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"]
Oct 31 00:24:40.408723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.408700    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.463919 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.463894    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"kube-root-ca.crt\""
Oct 31 00:24:40.464098 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.463993    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"distinct-2-w1-869cca-kserve-self-signed-certs\""
Oct 31 00:24:40.466428 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.466402    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"openshift-service-ca.crt\""
Oct 31 00:24:40.516012 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.515983    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"]
Oct 31 00:24:40.519075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.519051    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.519165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.519107    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.519219 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.519198    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.519348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.519330    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.519411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.519365    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.519501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.519481    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rsmm\" (UniqueName: \"kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.564718 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.564685    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-869cca\"/\"default-dockercfg-vgb7c\""
Oct 31 00:24:40.621111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621076    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621127    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621154    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621209    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621243    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621281    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rsmm\" (UniqueName: \"kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621582    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621657    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.621742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.621705    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.623845 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.623827    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.623984 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.623969    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.682477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.682451    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rsmm\" (UniqueName: \"kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm\") pod \"distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") " pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:40.720952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:40.720925    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:41.097118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:41.097091    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"]
Oct 31 00:24:41.099271 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:24:41.099246    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6d92ea20_9c90_4a1b_a2d6_5ba98df509d2.slice/crio-7c259b25109949a61b49c23f3e4d3d0603a69a4f37e7ab94ff860f3a56ba9a97 WatchSource:0}: Error finding container 7c259b25109949a61b49c23f3e4d3d0603a69a4f37e7ab94ff860f3a56ba9a97: Status 404 returned error can't find the container with id 7c259b25109949a61b49c23f3e4d3d0603a69a4f37e7ab94ff860f3a56ba9a97
Oct 31 00:24:42.023973 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:42.023940    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" event={"ID":"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2","Type":"ContainerStarted","Data":"ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78"}
Oct 31 00:24:42.024352 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:42.023981    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" event={"ID":"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2","Type":"ContainerStarted","Data":"7c259b25109949a61b49c23f3e4d3d0603a69a4f37e7ab94ff860f3a56ba9a97"}
Oct 31 00:24:42.141799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:42.141740    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podStartSLOduration=2.141723981 podStartE2EDuration="2.141723981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:42.140708182 +0000 UTC m=+1315.157372896" watchObservedRunningTime="2026-10-31 00:24:42.141723981 +0000 UTC m=+1315.158388693"
Oct 31 00:24:42.721512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:42.721473    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:44.758821 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:24:44.758776    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:24:50.723567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:50.723536    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:50.957404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:50.957203    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:51.149299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:51.149207    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 00:24:53.532406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:53.530526    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"]
Oct 31 00:24:58.531269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.531209    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld" podUID="251ce9dd-a54e-474f-b841-c54556186e0a" containerName="payload-processing" containerID="cri-o://5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f" gracePeriod=25
Oct 31 00:24:58.793242 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.793184    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:24:58.876837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.876809    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials\") pod \"251ce9dd-a54e-474f-b841-c54556186e0a\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") "
Oct 31 00:24:58.876993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.876852    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume\") pod \"251ce9dd-a54e-474f-b841-c54556186e0a\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") "
Oct 31 00:24:58.876993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.876977    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume\") pod \"251ce9dd-a54e-474f-b841-c54556186e0a\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") "
Oct 31 00:24:58.877107 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.877021    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg\") pod \"251ce9dd-a54e-474f-b841-c54556186e0a\" (UID: \"251ce9dd-a54e-474f-b841-c54556186e0a\") "
Oct 31 00:24:58.877341 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.877316    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume" pod "251ce9dd-a54e-474f-b841-c54556186e0a" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:58.877341 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.877328    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume" pod "251ce9dd-a54e-474f-b841-c54556186e0a" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:24:58.877511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.877495    4478 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-routing-overlay-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:24:58.877557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.877516    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/251ce9dd-a54e-474f-b841-c54556186e0a-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:24:58.879328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.879301    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg" pod "251ce9dd-a54e-474f-b841-c54556186e0a" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a"). InnerVolumeSpecName "kube-api-access-bpxdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:58.879328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.879322    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials" pod "251ce9dd-a54e-474f-b841-c54556186e0a" (UID: "251ce9dd-a54e-474f-b841-c54556186e0a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:24:58.979513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.979482    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpxdg\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-kube-api-access-bpxdg\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:24:58.979513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:58.979515    4478 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/251ce9dd-a54e-474f-b841-c54556186e0a-provider-credentials\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:24:59.177799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.177763    4478 generic.go:356] "Generic (PLEG): container finished" podID="251ce9dd-a54e-474f-b841-c54556186e0a" containerID="5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f" exitCode=0
Oct 31 00:24:59.177950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.177817    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"
Oct 31 00:24:59.177950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.177850    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld" event={"ID":"251ce9dd-a54e-474f-b841-c54556186e0a","Type":"ContainerDied","Data":"5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f"}
Oct 31 00:24:59.177950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.177905    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7cd8774477-vdcld" event={"ID":"251ce9dd-a54e-474f-b841-c54556186e0a","Type":"ContainerDied","Data":"f8027965126600d3ff237872eb40974725e7d1231c35ce75fb84316f9f7b7c6f"}
Oct 31 00:24:59.177950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.177928    4478 scope.go:122] "RemoveContainer" containerID="5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f"
Oct 31 00:24:59.186699 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.186676    4478 scope.go:122] "RemoveContainer" containerID="5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f"
Oct 31 00:24:59.186975 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:24:59.186955    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f\": container with ID starting with 5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f not found: ID does not exist" containerID="5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f"
Oct 31 00:24:59.187034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.186983    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f"} err="failed to get container status \"5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f\": rpc error: code = NotFound desc = could not find container \"5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f\": container with ID starting with 5a27b4fe9c731a3b1458d98ed62171512439276daaf7af6395cdc75a0948b62f not found: ID does not exist"
Oct 31 00:24:59.341991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.341960    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"]
Oct 31 00:24:59.391434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:24:59.391360    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7cd8774477-vdcld"]
Oct 31 00:25:01.116893 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:01.116862    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251ce9dd-a54e-474f-b841-c54556186e0a" path="/var/lib/kubelet/pods/251ce9dd-a54e-474f-b841-c54556186e0a/volumes"
Oct 31 00:25:15.089406 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:25:15.087622    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:25:44.757720 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:25:44.757684    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:25:57.611998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.611969    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"]
Oct 31 00:25:57.612465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.612452    4478 cpu_manager.go:414] "containerMap: removing container" podUID="251ce9dd-a54e-474f-b841-c54556186e0a" containerName="payload-processing"
Oct 31 00:25:57.612465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.612466    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251ce9dd-a54e-474f-b841-c54556186e0a" containerName="payload-processing"
Oct 31 00:25:57.612551 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.612546    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="251ce9dd-a54e-474f-b841-c54556186e0a" containerName="payload-processing"
Oct 31 00:25:57.616925 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.616909    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.668780 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.668750    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.668988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.668827    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gjvn\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.668988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.668947    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.669149 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.669007    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.676279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.676253    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"]
Oct 31 00:25:57.722906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.722862    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a2f1ffbe7a37\""
Oct 31 00:25:57.722906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.722892    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-h8pxf\""
Oct 31 00:25:57.723068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.722881    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 31 00:25:57.723068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.722867    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 31 00:25:57.724752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.724735    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 31 00:25:57.724816 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.724734    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 31 00:25:57.770667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.770642    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.770811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.770724    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gjvn\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.770811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.770790    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.770933 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.770850    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.771576 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.771504    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.771576 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.771560    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.773331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.773305    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.831422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.831368    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gjvn\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn\") pod \"payload-processing-external-model-7bcdc9f9cc-bkxqq\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") " pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:57.934759 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:57.934731    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:58.367932 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:58.367872    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"]
Oct 31 00:25:58.370721 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:25:58.370693    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f961884_f450_43c0_b10e_7d739b66f967.slice/crio-9c8ca051ec284fbb56eeb496489808830068c4882df1dda265f2e2584e3a6662 WatchSource:0}: Error finding container 9c8ca051ec284fbb56eeb496489808830068c4882df1dda265f2e2584e3a6662: Status 404 returned error can't find the container with id 9c8ca051ec284fbb56eeb496489808830068c4882df1dda265f2e2584e3a6662
Oct 31 00:25:58.836204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:58.836166    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" event={"ID":"6f961884-f450-43c0-b10e-7d739b66f967","Type":"ContainerStarted","Data":"607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085"}
Oct 31 00:25:58.836715 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:58.836234    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" event={"ID":"6f961884-f450-43c0-b10e-7d739b66f967","Type":"ContainerStarted","Data":"9c8ca051ec284fbb56eeb496489808830068c4882df1dda265f2e2584e3a6662"}
Oct 31 00:25:58.836715 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:58.836297    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:25:58.953841 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:25:58.953782    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" podStartSLOduration=1.9537678280000002 podStartE2EDuration="1.953767828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:58.951736513 +0000 UTC m=+1391.968401225" watchObservedRunningTime="2026-10-31 00:25:58.953767828 +0000 UTC m=+1391.970432540"
Oct 31 00:26:02.307518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.307486    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"]
Oct 31 00:26:02.311617 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.311598    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.426711 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.426677    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.426884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.426724    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.426884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.426842    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.426884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.426873    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.427010 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.426989    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.427072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.427057    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.427118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.427087    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9jvw\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.427118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.427107    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.427218 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.427195    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.519337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.519283    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"]
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529070    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529136    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9jvw\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529168    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529203    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529239    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529274    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529342    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529369    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.529496    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.538982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.530335    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.540583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.540555    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.540752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.540730    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.540856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.540799    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.540980 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.540957    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.546359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.546327    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.548646 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.548623    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.590238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.590175    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.590417 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.590395    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9jvw\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:02.625407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:02.625365    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:03.059125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:03.059101    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"]
Oct 31 00:26:03.061185 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:26:03.061155    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde645620_02e8_42ee_8687_02b4892cdc1c.slice/crio-4b5d8767cc8721a2a0f13d5d5282c5531b9b6230f4c3a43ebc160216684d2c13 WatchSource:0}: Error finding container 4b5d8767cc8721a2a0f13d5d5282c5531b9b6230f4c3a43ebc160216684d2c13: Status 404 returned error can't find the container with id 4b5d8767cc8721a2a0f13d5d5282c5531b9b6230f4c3a43ebc160216684d2c13
Oct 31 00:26:03.063372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:03.063338    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:03.063481 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:03.063437    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:03.063525 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:03.063501    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:03.871956 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:03.871915    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" event={"ID":"de645620-02e8-42ee-8687-02b4892cdc1c","Type":"ContainerStarted","Data":"8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd"}
Oct 31 00:26:03.872431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:03.871982    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" event={"ID":"de645620-02e8-42ee-8687-02b4892cdc1c","Type":"ContainerStarted","Data":"4b5d8767cc8721a2a0f13d5d5282c5531b9b6230f4c3a43ebc160216684d2c13"}
Oct 31 00:26:04.053407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:04.050909    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podStartSLOduration=2.050888024 podStartE2EDuration="2.050888024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:04.048878195 +0000 UTC m=+1397.065542938" watchObservedRunningTime="2026-10-31 00:26:04.050888024 +0000 UTC m=+1397.067552737"
Oct 31 00:26:04.626395 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:04.626348    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:04.627580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:04.627555    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:26:04.627689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:04.627604    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 00:26:05.626261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:05.626225    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:26:05.626662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:05.626280    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 00:26:06.626065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:06.626029    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:26:06.626238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:06.626086    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 00:26:07.626080 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:07.626045    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:26:07.626595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:07.626118    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 00:26:08.625953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:08.625922    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:26:08.626201 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:08.626176    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 00:26:09.626669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:09.626639    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused" start-of-body=
Oct 31 00:26:09.627188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:09.626689    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.66:15021/healthz/ready\": dial tcp 10.132.0.66:15021: connect: connection refused"
Oct 31 00:26:09.845608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:09.845565    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" podUID="6f961884-f450-43c0-b10e-7d739b66f967" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:26:10.440182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:10.440147    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"]
Oct 31 00:26:10.441559 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:10.441524    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" podUID="6f961884-f450-43c0-b10e-7d739b66f967" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:26:10.630923 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:10.630893    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:10.631360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:10.631181    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:10.631839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:10.631821    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:26:14.758956 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:26:14.758912    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:26:15.401590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.401555    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"]
Oct 31 00:26:15.405322 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.405305    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.440540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.440510    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" podUID="6f961884-f450-43c0-b10e-7d739b66f967" containerName="payload-processing" containerID="cri-o://607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085" gracePeriod=25
Oct 31 00:26:15.457600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.457578    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-0144bd-openshift-default-dockercfg-4hs4v\""
Oct 31 00:26:15.510741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.510714    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"]
Oct 31 00:26:15.523698 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.523659    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.523835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.523712    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.523835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.523815    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.523964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.523868    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.523964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.523923    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.524076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.523964    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.524076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.524005    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.524076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.524033    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g4k6\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.524228 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.524074    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625114    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625166    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625482    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625544 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625522    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625598 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625572    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625588    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625688    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625734    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625747    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625786    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625827    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.625953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.625858    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g4k6\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.626347 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.626317    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.627211 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.627181    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.628675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.628653    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.630163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.630139    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.687126 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.687098    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.687528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.687509    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g4k6\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.714909 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.714886    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:26:15.719171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.719155    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:15.827929 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.827901    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume\") pod \"6f961884-f450-43c0-b10e-7d739b66f967\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") "
Oct 31 00:26:15.828367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.828052    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials\") pod \"6f961884-f450-43c0-b10e-7d739b66f967\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") "
Oct 31 00:26:15.828367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.828098    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume\") pod \"6f961884-f450-43c0-b10e-7d739b66f967\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") "
Oct 31 00:26:15.828367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.828141    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn\") pod \"6f961884-f450-43c0-b10e-7d739b66f967\" (UID: \"6f961884-f450-43c0-b10e-7d739b66f967\") "
Oct 31 00:26:15.829247 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.828925    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume" pod "6f961884-f450-43c0-b10e-7d739b66f967" (UID: "6f961884-f450-43c0-b10e-7d739b66f967"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:15.829247 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.828959    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume" pod "6f961884-f450-43c0-b10e-7d739b66f967" (UID: "6f961884-f450-43c0-b10e-7d739b66f967"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:26:15.831967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.831938    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials" pod "6f961884-f450-43c0-b10e-7d739b66f967" (UID: "6f961884-f450-43c0-b10e-7d739b66f967"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:15.834119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.834079    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn" pod "6f961884-f450-43c0-b10e-7d739b66f967" (UID: "6f961884-f450-43c0-b10e-7d739b66f967"). InnerVolumeSpecName "kube-api-access-4gjvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:26:15.929799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.929727    4478 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-provider-credentials\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:26:15.929799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.929753    4478 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-routing-overlay-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:26:15.929799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.929763    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gjvn\" (UniqueName: \"kubernetes.io/projected/6f961884-f450-43c0-b10e-7d739b66f967-kube-api-access-4gjvn\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:26:15.929799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.929772    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f961884-f450-43c0-b10e-7d739b66f967-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:26:15.947658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.947625    4478 generic.go:356] "Generic (PLEG): container finished" podID="6f961884-f450-43c0-b10e-7d739b66f967" containerID="607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085" exitCode=0
Oct 31 00:26:15.947833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.947667    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"
Oct 31 00:26:15.947833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.947692    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" event={"ID":"6f961884-f450-43c0-b10e-7d739b66f967","Type":"ContainerDied","Data":"607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085"}
Oct 31 00:26:15.947833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.947745    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq" event={"ID":"6f961884-f450-43c0-b10e-7d739b66f967","Type":"ContainerDied","Data":"9c8ca051ec284fbb56eeb496489808830068c4882df1dda265f2e2584e3a6662"}
Oct 31 00:26:15.947833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.947776    4478 scope.go:122] "RemoveContainer" containerID="607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085"
Oct 31 00:26:15.959686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.959659    4478 scope.go:122] "RemoveContainer" containerID="607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085"
Oct 31 00:26:15.960026 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:26:15.959993    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085\": container with ID starting with 607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085 not found: ID does not exist" containerID="607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085"
Oct 31 00:26:15.960090 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:15.960037    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085"} err="failed to get container status \"607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085\": rpc error: code = NotFound desc = could not find container \"607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085\": container with ID starting with 607845532ad6079abbedce8a6804c18f159747590970ae2274ee1b16904b1085 not found: ID does not exist"
Oct 31 00:26:16.115521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.115490    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"]
Oct 31 00:26:16.121687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.121659    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-7bcdc9f9cc-bkxqq"]
Oct 31 00:26:16.160877 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.160855    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"]
Oct 31 00:26:16.164253 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:26:16.164224    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67adc0b3_30aa_4d12_99b7_860afd15ba60.slice/crio-f49edda89331cf5749eebc6da724242ccdd4eb9bbfc8ade3b72377a283205993 WatchSource:0}: Error finding container f49edda89331cf5749eebc6da724242ccdd4eb9bbfc8ade3b72377a283205993: Status 404 returned error can't find the container with id f49edda89331cf5749eebc6da724242ccdd4eb9bbfc8ade3b72377a283205993
Oct 31 00:26:16.166725 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.166691    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:16.166811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.166760    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:16.166811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.166799    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:16.961882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.960925    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" event={"ID":"67adc0b3-30aa-4d12-99b7-860afd15ba60","Type":"ContainerStarted","Data":"bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c"}
Oct 31 00:26:16.961882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:16.960995    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" event={"ID":"67adc0b3-30aa-4d12-99b7-860afd15ba60","Type":"ContainerStarted","Data":"f49edda89331cf5749eebc6da724242ccdd4eb9bbfc8ade3b72377a283205993"}
Oct 31 00:26:17.104999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:17.104934    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podStartSLOduration=2.104914421 podStartE2EDuration="2.104914421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:17.101762376 +0000 UTC m=+1410.118427091" watchObservedRunningTime="2026-10-31 00:26:17.104914421 +0000 UTC m=+1410.121579134"
Oct 31 00:26:17.124487 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:17.124451    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f961884-f450-43c0-b10e-7d739b66f967" path="/var/lib/kubelet/pods/6f961884-f450-43c0-b10e-7d739b66f967/volumes"
Oct 31 00:26:17.734649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:17.734623    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:17.810565 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:17.810528    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:17.814486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:17.814451    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:18.745408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:18.729617    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:18.745408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:18.729681    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:19.998079 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:19.983828    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:19.999162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:19.998741    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:20.127112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:20.122309    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:20.751407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:20.750488    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:20.751407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:20.750565    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:21.725875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:21.725828    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:21.726494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:21.726451    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:21.742058 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:21.741959    4478 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:26:22.155560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.155459    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 00:26:22.720408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.719869    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:22.720408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.719961    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:22.849162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.849125    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"]
Oct 31 00:26:22.850610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.850582    4478 cpu_manager.go:414] "containerMap: removing container" podUID="6f961884-f450-43c0-b10e-7d739b66f967" containerName="payload-processing"
Oct 31 00:26:22.850775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.850752    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f961884-f450-43c0-b10e-7d739b66f967" containerName="payload-processing"
Oct 31 00:26:22.851047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.851031    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f961884-f450-43c0-b10e-7d739b66f967" containerName="payload-processing"
Oct 31 00:26:22.880680 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.880650    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:22.924405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.922742    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"]
Oct 31 00:26:22.931413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.928638    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:22.983451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:22.981252    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"]
Oct 31 00:26:23.009439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.004796    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"]
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095419    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095510    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjsx6\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095538    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095563    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095602    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095633    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095657    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095683    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095713    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095741    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095768    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095817    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095840    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095875    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.095943    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.096003    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.096036    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.102401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.096064    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58bfn\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.203757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.203716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.203944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.203801    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.203944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.203838    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.203944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.203870    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58bfn\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.203961    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204040    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjsx6\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204067    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204122 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204094    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204138    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204175    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204205    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204233    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204269    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204301    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204330    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.204719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204365    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204411    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.204719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.204452    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.205233 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.205200    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.205579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.205554    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.206093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.206072    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.206216 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.206159    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.206450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.206422    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.207853 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.207289    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.207853 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.207319    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.207853 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.207594    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.208068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.207988    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.209413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.208634    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.211976 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.211948    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.212483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.212402    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.212873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.212849    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.213875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.213852    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.282447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.282251    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.292520 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.291839    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.292520 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.291957    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjsx6\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.292520 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.292455    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58bfn\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn\") pod \"e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") " pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.517926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.517883    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:23.578622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.578539    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:23.719910 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.719874    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:23.720059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:23.719924    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:24.020169 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.020003    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"]
Oct 31 00:26:24.022278 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:26:24.022252    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e36ff2_88d8_4c0a_8c03_c8271fcf4c36.slice/crio-07c305473e7037eab38561da9ec2b81d7fd22f0e2731062ea5651c420c541dec WatchSource:0}: Error finding container 07c305473e7037eab38561da9ec2b81d7fd22f0e2731062ea5651c420c541dec: Status 404 returned error can't find the container with id 07c305473e7037eab38561da9ec2b81d7fd22f0e2731062ea5651c420c541dec
Oct 31 00:26:24.022421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.022362    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"]
Oct 31 00:26:24.024603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.024572    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:24.024699 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.024644    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:24.024699 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.024688    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:24.081658 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:26:24.081615    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa1075b1_41b0_4857_8701_657c5332c744.slice/crio-d65eabea9c754a5f929067b3c580fa4c177b7593f58fdd09f7da60e823b6c5de WatchSource:0}: Error finding container d65eabea9c754a5f929067b3c580fa4c177b7593f58fdd09f7da60e823b6c5de: Status 404 returned error can't find the container with id d65eabea9c754a5f929067b3c580fa4c177b7593f58fdd09f7da60e823b6c5de
Oct 31 00:26:24.084516 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.084480    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:24.084607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.084573    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:24.084650 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.084632    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:24.306840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.306745    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" event={"ID":"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36","Type":"ContainerStarted","Data":"7650c165190f3fdb24d25c8c68419b23098326e02677d37f0836cb32b454af74"}
Oct 31 00:26:24.306840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.306816    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" event={"ID":"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36","Type":"ContainerStarted","Data":"07c305473e7037eab38561da9ec2b81d7fd22f0e2731062ea5651c420c541dec"}
Oct 31 00:26:24.309358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.309326    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" event={"ID":"aa1075b1-41b0-4857-8701-657c5332c744","Type":"ContainerStarted","Data":"05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88"}
Oct 31 00:26:24.309500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.309403    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" event={"ID":"aa1075b1-41b0-4857-8701-657c5332c744","Type":"ContainerStarted","Data":"d65eabea9c754a5f929067b3c580fa4c177b7593f58fdd09f7da60e823b6c5de"}
Oct 31 00:26:24.430487 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.430414    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podStartSLOduration=2.430395276 podStartE2EDuration="2.430395276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:24.428795942 +0000 UTC m=+1417.445460654" watchObservedRunningTime="2026-10-31 00:26:24.430395276 +0000 UTC m=+1417.447059980"
Oct 31 00:26:24.519090 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.519053    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:24.579286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.579194    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:24.720576 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.720543    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:24.720737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:24.720594    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:25.519627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:25.519590    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:26:25.520025 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:25.519651    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 00:26:25.583840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:25.583803    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:26:25.584151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:25.584109    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 00:26:25.720561 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:25.720529    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Oct 31 00:26:25.720720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:25.720581    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Oct 31 00:26:26.519470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.519433    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:26:26.519669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.519492    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 00:26:26.579858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.579820    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:26:26.580173 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.580136    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 00:26:26.724824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.724797    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:26.725808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.725652    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:26.726249 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.726228    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:26:26.889162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:26.889029    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podStartSLOduration=4.889008713 podStartE2EDuration="4.889008713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:24.556876897 +0000 UTC m=+1417.573541611" watchObservedRunningTime="2026-10-31 00:26:26.889008713 +0000 UTC m=+1419.905673427"
Oct 31 00:26:27.518635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:27.518552    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:26:27.518635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:27.518624    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 00:26:27.585217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:27.585178    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:26:27.585854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:27.585825    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 00:26:28.519407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:28.519211    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:26:28.519407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:28.519292    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 00:26:28.584408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:28.580729    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:26:28.584408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:28.580791    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 00:26:29.518679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:29.518644    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:26:29.519118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:29.518695    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 00:26:29.580040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:29.580006    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:26:29.580345 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:29.580302    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 00:26:30.518537 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:30.518494    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body=
Oct 31 00:26:30.518716 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:30.518553    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused"
Oct 31 00:26:30.579770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:30.579738    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Oct 31 00:26:30.580010 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:30.579983    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Oct 31 00:26:31.784027 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:31.783987    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:31.785041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:31.785014    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:31.786147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:31.786123    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:26:31.837346 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:31.837313    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:31.837725 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:31.837704    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:31.839457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:31.839438    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:26:32.620653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.620614    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"]
Oct 31 00:26:32.624835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.624816    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.713578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.713536    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"]
Oct 31 00:26:32.742091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742055    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742184    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqbg\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742253    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742302    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742452    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742490    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742521    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742573    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.742618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.742612    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844318    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844363    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844435    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844471    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844519    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844620    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844675    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqbg\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844727    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844729    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844797    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844820    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.844914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.844915    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.845284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.845030    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.845437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.845416    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.847077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.847049    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.847245 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.847229    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.905342 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.905273    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.905342 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.905334    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqbg\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg\") pod \"e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") " pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:32.938367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:32.938345    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:33.379111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:33.379088    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"]
Oct 31 00:26:33.380892 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:26:33.380864    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65663a9c_63df_4a76_ad07_26b7a1597b87.slice/crio-ab18b2df76b19c7bec392968717a8163af030496aa4e2127ad96cf6a261bf20b WatchSource:0}: Error finding container ab18b2df76b19c7bec392968717a8163af030496aa4e2127ad96cf6a261bf20b: Status 404 returned error can't find the container with id ab18b2df76b19c7bec392968717a8163af030496aa4e2127ad96cf6a261bf20b
Oct 31 00:26:33.383204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:33.383168    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:33.383328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:33.383255    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:33.383328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:33.383302    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:26:33.395214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:33.395186    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" event={"ID":"65663a9c-63df-4a76-ad07-26b7a1597b87","Type":"ContainerStarted","Data":"ab18b2df76b19c7bec392968717a8163af030496aa4e2127ad96cf6a261bf20b"}
Oct 31 00:26:34.402518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:34.402479    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" event={"ID":"65663a9c-63df-4a76-ad07-26b7a1597b87","Type":"ContainerStarted","Data":"ae5a544637257d896d4c0609869b42e96a4e875665c8e4f257c1037b25201bb7"}
Oct 31 00:26:34.624150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:34.624092    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podStartSLOduration=2.624076842 podStartE2EDuration="2.624076842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:34.572477939 +0000 UTC m=+1427.589142675" watchObservedRunningTime="2026-10-31 00:26:34.624076842 +0000 UTC m=+1427.640741553"
Oct 31 00:26:34.951405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:34.948703    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:34.961433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:34.959524    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:26:34.961433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:34.959617    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 00:26:35.939405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:35.938730    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:26:35.939405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:35.938802    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 00:26:36.951405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:36.942703    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:26:36.951405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:36.942773    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 00:26:37.945405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:37.943722    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:26:37.945405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:37.943781    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 00:26:38.950406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:38.949765    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:26:38.950406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:38.949830    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 00:26:39.938890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:39.938853    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body=
Oct 31 00:26:39.939059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:39.938921    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused"
Oct 31 00:26:40.942960 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:40.942928    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:40.943372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:40.943194    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:40.943892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:26:40.943877    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:26:44.783603 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:26:44.783568    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:27:00.960408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:00.952449    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"]
Oct 31 00:27:01.048405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.046602    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"]
Oct 31 00:27:01.048405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.046749    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.172712    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.172759    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.172793    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.172932    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.172980    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.173066    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.173093    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqv5h\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.173125    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.177405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.173174    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.410466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.400657    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469110    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqv5h\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469194    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469253    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469337    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469366    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469414    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469564    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.469628    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.470116    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.403082    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.483403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.481589    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.496675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.496639    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.497609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.497580    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.601409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.596830    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.653215 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.644295    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.736335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.736297    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqv5h\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.737140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.737054    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token\") pod \"e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") " pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:01.802928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:01.802894    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:02.140408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:02.139512    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:02.917460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:02.916003    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"]
Oct 31 00:27:03.035620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.027810    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"]
Oct 31 00:27:03.035620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.027978    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.153854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.153816    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.153865    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.153894    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.153964    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.153998    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.154025    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.154163    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f848l\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.154203    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.154337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.154241    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.273698    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f848l\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.273762    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.273804    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.273887    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.273917    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.273948    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.274027    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.274060    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.274500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.274085    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.275140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.275023    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.294407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.275617    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.294407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.275871    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.294407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.276112    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.330407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.318810    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.355489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.334075    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"]
Oct 31 00:27:03.397451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.373330    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.449050 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.415857    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.482781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.482166    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.511654 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.511518    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:27:03.512011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.511943    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:27:03.512188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.512160    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:27:03.514783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.514749    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" event={"ID":"c33bf3fd-0d13-4b7e-bb51-b11249184aba","Type":"ContainerStarted","Data":"a8958978c8ed9ceb48060413a3fd05c4e41dd2cf6ceb6f381ded30296a0d04b6"}
Oct 31 00:27:03.556718 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.538360    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f848l\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:03.729235 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:03.729203    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:04.306904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.306872    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"]
Oct 31 00:27:04.308630 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:27:04.308605    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb169c664_7798_4ef4_bdfa_efbadc4ccfd4.slice/crio-2eeedddea42b199fa2b54dba7068724bcd8d879d82bab9fb231ed97c05b3984a WatchSource:0}: Error finding container 2eeedddea42b199fa2b54dba7068724bcd8d879d82bab9fb231ed97c05b3984a: Status 404 returned error can't find the container with id 2eeedddea42b199fa2b54dba7068724bcd8d879d82bab9fb231ed97c05b3984a
Oct 31 00:27:04.311066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.311034    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:27:04.311164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.311100    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:27:04.311164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.311129    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:27:04.529213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.528086    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" event={"ID":"c33bf3fd-0d13-4b7e-bb51-b11249184aba","Type":"ContainerStarted","Data":"098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098"}
Oct 31 00:27:04.532667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.531832    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" event={"ID":"b169c664-7798-4ef4-bdfa-efbadc4ccfd4","Type":"ContainerStarted","Data":"ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e"}
Oct 31 00:27:04.532667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.531895    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" event={"ID":"b169c664-7798-4ef4-bdfa-efbadc4ccfd4","Type":"ContainerStarted","Data":"2eeedddea42b199fa2b54dba7068724bcd8d879d82bab9fb231ed97c05b3984a"}
Oct 31 00:27:04.670588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.670464    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podStartSLOduration=4.670441038 podStartE2EDuration="4.670441038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:04.652769652 +0000 UTC m=+1457.669434364" watchObservedRunningTime="2026-10-31 00:27:04.670441038 +0000 UTC m=+1457.687105751"
Oct 31 00:27:04.729619 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.729588    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:04.804754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.804723    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:04.806039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.806016    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:04.806160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:04.806073    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:05.738418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:05.735311    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:05.738418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:05.735375    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:05.815407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:05.815205    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:05.815407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:05.815265    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:06.731048 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:06.731011    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:06.731228 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:06.731074    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:06.804766 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:06.804732    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:06.805151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:06.804783    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:07.730440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:07.730409    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:07.730614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:07.730468    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:07.804310 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:07.804278    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:07.804497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:07.804341    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:08.743408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:08.734765    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:08.743408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:08.734832    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:08.876753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:08.876105    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:08.876753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:08.876176    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:09.134328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.131265    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podStartSLOduration=7.131242565 podStartE2EDuration="7.131242565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:04.780725642 +0000 UTC m=+1457.797390363" watchObservedRunningTime="2026-10-31 00:27:09.131242565 +0000 UTC m=+1462.147907279"
Oct 31 00:27:09.450744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.450712    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"]
Oct 31 00:27:09.528658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.528631    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"]
Oct 31 00:27:09.528920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.528903    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"]
Oct 31 00:27:09.529895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.528756    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.553890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.553858    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"]
Oct 31 00:27:09.590129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.555139    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.591128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.591106    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f7c92a85\""
Oct 31 00:27:09.592153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.591465    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f7c92a85-dockercfg-55xzp\""
Oct 31 00:27:09.678450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.678415    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.678807 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.678784    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq6pd\" (UniqueName: \"kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.678970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.678953    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.679178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.679162    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njwnp\" (UniqueName: \"kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.730592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.730517    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:09.730793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.730765    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:09.788421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.787713    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.789041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.788921    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.789171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.789033    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq6pd\" (UniqueName: \"kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.789286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.789270    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.789478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.789462    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njwnp\" (UniqueName: \"kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.792429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.790725    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.805966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.805932    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:09.806082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.805981    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:09.874409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.872222    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njwnp\" (UniqueName: \"kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp\") pod \"payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:09.899336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.898153    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq6pd\" (UniqueName: \"kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.899336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.898354    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:09.907424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:09.907297    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:10.387679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.381610    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"]
Oct 31 00:27:10.410275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.410249    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"]
Oct 31 00:27:10.410763 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:27:10.410738    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69da6e4c_5565_405d_92ea_75815f60ee05.slice/crio-946107ee0351488cab118c28a50c9c5cd0968cb69a6dcbb74dbcc9ea48dda286 WatchSource:0}: Error finding container 946107ee0351488cab118c28a50c9c5cd0968cb69a6dcbb74dbcc9ea48dda286: Status 404 returned error can't find the container with id 946107ee0351488cab118c28a50c9c5cd0968cb69a6dcbb74dbcc9ea48dda286
Oct 31 00:27:10.609317 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.609281    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" event={"ID":"fbd12343-79c6-48e7-9e9c-fc7abe167962","Type":"ContainerStarted","Data":"211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9"}
Oct 31 00:27:10.609514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.609341    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" event={"ID":"fbd12343-79c6-48e7-9e9c-fc7abe167962","Type":"ContainerStarted","Data":"06737ff6f023d9c39534a03653a8d30569fb1b5a5eab8f3328dbceec7ed3d5a1"}
Oct 31 00:27:10.609514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.609504    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:10.611094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.611070    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" event={"ID":"69da6e4c-5565-405d-92ea-75815f60ee05","Type":"ContainerStarted","Data":"71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352"}
Oct 31 00:27:10.611213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.611114    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" event={"ID":"69da6e4c-5565-405d-92ea-75815f60ee05","Type":"ContainerStarted","Data":"946107ee0351488cab118c28a50c9c5cd0968cb69a6dcbb74dbcc9ea48dda286"}
Oct 31 00:27:10.611213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.611161    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:10.729692 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.729657    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:10.729855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.729718    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:10.730651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.730603    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" podStartSLOduration=1.730586586 podStartE2EDuration="1.730586586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:10.727306505 +0000 UTC m=+1463.743971219" watchObservedRunningTime="2026-10-31 00:27:10.730586586 +0000 UTC m=+1463.747251301"
Oct 31 00:27:10.805024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.804983    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 31 00:27:10.805518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.805043    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 31 00:27:10.855406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:10.848116    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" podStartSLOduration=1.848097511 podStartE2EDuration="1.848097511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:10.844689886 +0000 UTC m=+1463.861354600" watchObservedRunningTime="2026-10-31 00:27:10.848097511 +0000 UTC m=+1463.864762224"
Oct 31 00:27:11.729881 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:11.729826    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body=
Oct 31 00:27:11.730151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:11.729900    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused"
Oct 31 00:27:12.061082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:12.060990    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:12.061573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:12.061514    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:12.062305 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:12.062279    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:27:12.734734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:12.734706    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:12.735185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:12.735172    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:12.736135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:12.736102    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:27:14.759858 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:27:14.759823    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:27:16.097754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.097716    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"]
Oct 31 00:27:16.102636 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.102609    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.183974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.183945    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.184136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.183991    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.184180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.184130    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkhnf\" (UniqueName: \"kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.184215 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.184195    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.184276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.184254    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.184416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.184399    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.210261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.210222    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"openshift-service-ca.crt\""
Oct 31 00:27:16.210483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.210233    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"default-dockercfg-2clwg\""
Oct 31 00:27:16.210483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.210361    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"distinct-2-w0-10a7aa-kserve-self-signed-certs\""
Oct 31 00:27:16.210483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.210401    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-10a7aa\"/\"kube-root-ca.crt\""
Oct 31 00:27:16.213546 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.213525    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"]
Oct 31 00:27:16.293176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.293139    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkhnf\" (UniqueName: \"kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.293552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.293513    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.293786 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.293749    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.294158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.294122    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.294340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.294324    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.294632 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.294606    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.296581 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.296367    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.298510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.298482    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.301246 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.301227    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.357433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.345278    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.357433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.350447    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.359559 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.359523    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkhnf\" (UniqueName: \"kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf\") pod \"distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") " pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.419331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.419298    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:16.873675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:16.873647    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"]
Oct 31 00:27:16.875608 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:27:16.875574    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5f8009e_242f_45b3_a85e_c7faf4550aa4.slice/crio-b1352008e7722b5bf35d0fa66fe13b20def5d4fb06fa433a1e1cf49e689c53c3 WatchSource:0}: Error finding container b1352008e7722b5bf35d0fa66fe13b20def5d4fb06fa433a1e1cf49e689c53c3: Status 404 returned error can't find the container with id b1352008e7722b5bf35d0fa66fe13b20def5d4fb06fa433a1e1cf49e689c53c3
Oct 31 00:27:17.670557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:17.670516    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" event={"ID":"c5f8009e-242f-45b3-a85e-c7faf4550aa4","Type":"ContainerStarted","Data":"c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4"}
Oct 31 00:27:17.671101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:17.670569    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" event={"ID":"c5f8009e-242f-45b3-a85e-c7faf4550aa4","Type":"ContainerStarted","Data":"b1352008e7722b5bf35d0fa66fe13b20def5d4fb06fa433a1e1cf49e689c53c3"}
Oct 31 00:27:17.789398 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:17.789341    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" podStartSLOduration=1.789327404 podStartE2EDuration="1.789327404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:17.786297788 +0000 UTC m=+1470.802962515" watchObservedRunningTime="2026-10-31 00:27:17.789327404 +0000 UTC m=+1470.805992115"
Oct 31 00:27:18.419941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:18.419912    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:21.649442 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:21.646560    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:27:21.657408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:21.653546    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:27:22.120405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:22.100526    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:25.158404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:25.143499    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:27:25.158404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:25.143600    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:25.164976 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:25.164832    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:27:25.164976 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:25.164906    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:25.313415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:25.249463    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:27:25.313415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:25.249534    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:26.419715 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:26.419683    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:26.437013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:26.436979    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:27.377869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:27.377838    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 00:27:35.350142 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.350069    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"]
Oct 31 00:27:35.355103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.355087    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.431076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.431019    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.431247 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.431092    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82542\" (UniqueName: \"kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.431247 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.431170    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.431410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.431337    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.459536 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.459503    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-f7c92a85-serving-cert\""
Oct 31 00:27:35.464192 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.464164    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"]
Oct 31 00:27:35.532685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.532645    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.532863 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:27:35.532778    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-serving-cert: secret "maas-api-e2e-mt-f7c92a85-serving-cert" not found
Oct 31 00:27:35.532863 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.532775    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82542\" (UniqueName: \"kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.532863 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:27:35.532838    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls podName:7e8f1315-25e9-4b99-905f-72c5e925086d nodeName:}" failed. No retries permitted until 2026-10-31 00:27:36.032816953 +0000 UTC m=+1489.049481657 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls") pod "maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" (UID: "7e8f1315-25e9-4b99-905f-72c5e925086d") : secret "maas-api-e2e-mt-f7c92a85-serving-cert" not found
Oct 31 00:27:35.533038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.532899    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.533090 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.533039    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.533687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.533654    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.535677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.535648    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:35.643532 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:35.643456    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82542\" (UniqueName: \"kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:36.046334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:36.046300    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:36.049304 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:36.049271    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\") pod \"maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:36.270727 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:36.270694    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:36.649624 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:36.649592    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"]
Oct 31 00:27:36.651039 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:27:36.651010    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e8f1315_25e9_4b99_905f_72c5e925086d.slice/crio-9e3b3406e85d1ab6a473f61c85ac7e3ee21ad5898317a1cc27d5de2f82bd7c4d WatchSource:0}: Error finding container 9e3b3406e85d1ab6a473f61c85ac7e3ee21ad5898317a1cc27d5de2f82bd7c4d: Status 404 returned error can't find the container with id 9e3b3406e85d1ab6a473f61c85ac7e3ee21ad5898317a1cc27d5de2f82bd7c4d
Oct 31 00:27:36.653059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:36.653039    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:27:37.425745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:37.425715    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" event={"ID":"7e8f1315-25e9-4b99-905f-72c5e925086d","Type":"ContainerStarted","Data":"9e3b3406e85d1ab6a473f61c85ac7e3ee21ad5898317a1cc27d5de2f82bd7c4d"}
Oct 31 00:27:38.491639 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:38.491598    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" event={"ID":"7e8f1315-25e9-4b99-905f-72c5e925086d","Type":"ContainerStarted","Data":"5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791"}
Oct 31 00:27:38.493628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:38.492677    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:27:38.621439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:38.621375    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" podStartSLOduration=2.587618872 podStartE2EDuration="3.62136055s" totalImagesPullingTime="1.033741678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:38.619143376 +0000 UTC m=+1491.635808101" watchObservedRunningTime="2026-10-31 00:27:38.62136055 +0000 UTC m=+1491.638025261"
Oct 31 00:27:42.108407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:42.107503    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": context deadline exceeded"
Oct 31 00:27:42.378460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:42.377464    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:27:44.785769 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:27:44.785728    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:27:45.511907 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:27:45.511879    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:28:14.878407 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:28:14.877658    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:28:30.651483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.651446    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"]
Oct 31 00:28:30.655413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.655398    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.692865 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.692841    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtnf6\" (UniqueName: \"kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6\") pod \"payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.692992 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.692889    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.772405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.763512    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a4dc2e-dockercfg-9gkq7\""
Oct 31 00:28:30.772405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.763628    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a4dc2e\""
Oct 31 00:28:30.772405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.770657    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"]
Oct 31 00:28:30.803279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.803237    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtnf6\" (UniqueName: \"kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6\") pod \"payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.882622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.882589    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.890521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.888591    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.903327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.901054    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"]
Oct 31 00:28:30.937878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.937850    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"]
Oct 31 00:28:30.938181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.938165    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:30.942180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.942147    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtnf6\" (UniqueName: \"kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6\") pod \"payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.968880 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.968852    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:30.994183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.994148    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fz2l\" (UniqueName: \"kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l\") pod \"payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:30.994392 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:30.994236    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:31.095440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.095409    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fz2l\" (UniqueName: \"kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l\") pod \"payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:31.095767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.095745    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:31.096695 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.096667    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:31.156222 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.156164    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fz2l\" (UniqueName: \"kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l\") pod \"payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") " pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:31.277981 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.277954    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:31.380866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.380827    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"]
Oct 31 00:28:31.387423 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:28:31.387396    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1b72a5_b65d_495c_9a69_5e341c8c8973.slice/crio-62d498146e16bbc306d8ab7235dea456c886ef1509daaa1681ac9f0e30278d63 WatchSource:0}: Error finding container 62d498146e16bbc306d8ab7235dea456c886ef1509daaa1681ac9f0e30278d63: Status 404 returned error can't find the container with id 62d498146e16bbc306d8ab7235dea456c886ef1509daaa1681ac9f0e30278d63
Oct 31 00:28:31.477217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.477191    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz" event={"ID":"0c1b72a5-b65d-495c-9a69-5e341c8c8973","Type":"ContainerStarted","Data":"faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675"}
Oct 31 00:28:31.477319 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.477239    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz" event={"ID":"0c1b72a5-b65d-495c-9a69-5e341c8c8973","Type":"ContainerStarted","Data":"62d498146e16bbc306d8ab7235dea456c886ef1509daaa1681ac9f0e30278d63"}
Oct 31 00:28:31.477430 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.477414    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:31.592830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.592772    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz" podStartSLOduration=1.59275323 podStartE2EDuration="1.59275323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:31.591512423 +0000 UTC m=+1544.608177167" watchObservedRunningTime="2026-10-31 00:28:31.59275323 +0000 UTC m=+1544.609417944"
Oct 31 00:28:31.657931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:31.657909    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"]
Oct 31 00:28:31.659490 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:28:31.659462    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd47ce4eb_36e7_47f1_ae35_ef6188200464.slice/crio-c57f75e07ef5c429b452bf747f136608ec828e376352346feac99e2a6fad5ef8 WatchSource:0}: Error finding container c57f75e07ef5c429b452bf747f136608ec828e376352346feac99e2a6fad5ef8: Status 404 returned error can't find the container with id c57f75e07ef5c429b452bf747f136608ec828e376352346feac99e2a6fad5ef8
Oct 31 00:28:32.486959 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:32.486927    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk" event={"ID":"d47ce4eb-36e7-47f1-ae35-ef6188200464","Type":"ContainerStarted","Data":"eb4cc77384e2ed5dc8ad7ac6858e2ec745224fbcaeb47b3727820c60bff5ba6b"}
Oct 31 00:28:32.487128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:32.486982    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk" event={"ID":"d47ce4eb-36e7-47f1-ae35-ef6188200464","Type":"ContainerStarted","Data":"c57f75e07ef5c429b452bf747f136608ec828e376352346feac99e2a6fad5ef8"}
Oct 31 00:28:32.487128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:32.487004    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:32.602650 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:32.602601    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk" podStartSLOduration=2.602586422 podStartE2EDuration="2.602586422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:32.601767276 +0000 UTC m=+1545.618432007" watchObservedRunningTime="2026-10-31 00:28:32.602586422 +0000 UTC m=+1545.619251134"
Oct 31 00:28:42.491571 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:42.491530    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:28:43.495772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:43.495746    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:28:44.761068 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:28:44.761021    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:28:59.248616 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.248528    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"]
Oct 31 00:28:59.253610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.253587    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.331468 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.329292    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"]
Oct 31 00:28:59.358671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.358637    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a4dc2e-serving-cert\""
Oct 31 00:28:59.426110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.426077    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.426288 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.426226    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rsg\" (UniqueName: \"kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.426349 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.426324    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.426420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.426391    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.527244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.527160    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.527700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.527676    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rsg\" (UniqueName: \"kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.527910 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.527891    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.528049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.528031    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.528209 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:28:59.528188    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-serving-cert: secret "maas-api-e2e-api-a-a4dc2e-serving-cert" not found
Oct 31 00:28:59.528274 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:28:59.528267    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls podName:1b48f889-2599-481c-b356-cd3395edb2de nodeName:}" failed. No retries permitted until 2026-10-31 00:29:00.028246323 +0000 UTC m=+1573.044911027 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls") pod "maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" (UID: "1b48f889-2599-481c-b356-cd3395edb2de") : secret "maas-api-e2e-api-a-a4dc2e-serving-cert" not found
Oct 31 00:28:59.528726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.528699    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.530926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.530900    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:28:59.591651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:28:59.591615    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rsg\" (UniqueName: \"kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:29:00.051806 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:00.051766    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:29:00.054563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:00.054529    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\") pod \"maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:29:00.167371 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:00.167341    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:29:00.553288 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:29:00.553255    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b48f889_2599_481c_b356_cd3395edb2de.slice/crio-eb873078312fbbc96a5cf23cb1ee3f3e85a5b3ee07e56792995a6c10b9769453 WatchSource:0}: Error finding container eb873078312fbbc96a5cf23cb1ee3f3e85a5b3ee07e56792995a6c10b9769453: Status 404 returned error can't find the container with id eb873078312fbbc96a5cf23cb1ee3f3e85a5b3ee07e56792995a6c10b9769453
Oct 31 00:29:00.553934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:00.553908    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"]
Oct 31 00:29:00.662500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:00.662468    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" event={"ID":"1b48f889-2599-481c-b356-cd3395edb2de","Type":"ContainerStarted","Data":"eb873078312fbbc96a5cf23cb1ee3f3e85a5b3ee07e56792995a6c10b9769453"}
Oct 31 00:29:01.669640 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:01.669609    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" event={"ID":"1b48f889-2599-481c-b356-cd3395edb2de","Type":"ContainerStarted","Data":"d38e2378c6c5a4b580684df31f4f250673b8b4d96b9997292ecca5bc9df02b31"}
Oct 31 00:29:01.670076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:01.669760    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:29:01.787838 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:01.787738    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" podStartSLOduration=1.862660916 podStartE2EDuration="2.787724203s" totalImagesPullingTime="925.063287ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:01.78605565 +0000 UTC m=+1574.802720378" watchObservedRunningTime="2026-10-31 00:29:01.787724203 +0000 UTC m=+1574.804388914"
Oct 31 00:29:07.681854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:07.681826    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:29:14.762507 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:29:14.762453    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:29:16.925214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:16.925178    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"]
Oct 31 00:29:16.929894 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:16.929876    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.018294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018248    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.018480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018317    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.018480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018363    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.018480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018450    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.020458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018486    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.020458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018526    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6z4pj\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.020458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018563    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.020458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018767    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.020458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.018865    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.037371 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.037339    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"]
Oct 31 00:29:17.119555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119527    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119581    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119604    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119634    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119661    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119806    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119857    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119900    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6z4pj\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.119961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.119941    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.120196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.120173    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.120425 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.120406    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.120498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.120455    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.120590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.120564    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.120706 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.120602    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.122615 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.122594    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.122928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.122907    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.181147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.181081    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.181337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.181311    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6z4pj\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.243294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.243264    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:17.678811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.678773    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"]
Oct 31 00:29:17.681103 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:29:17.681073    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d11bad4_844a_428f_b295_b7500cdeb3d0.slice/crio-ac3e3ae9a1bee138edc2aa56bef1a6d733627dcbbb30d7a0c9e2a471e0e30e9f WatchSource:0}: Error finding container ac3e3ae9a1bee138edc2aa56bef1a6d733627dcbbb30d7a0c9e2a471e0e30e9f: Status 404 returned error can't find the container with id ac3e3ae9a1bee138edc2aa56bef1a6d733627dcbbb30d7a0c9e2a471e0e30e9f
Oct 31 00:29:17.683843 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.683806    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:29:17.683907 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.683877    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:29:17.683945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.683906    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:29:17.793345 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.793307    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" event={"ID":"1d11bad4-844a-428f-b295-b7500cdeb3d0","Type":"ContainerStarted","Data":"a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55"}
Oct 31 00:29:17.793567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.793364    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" event={"ID":"1d11bad4-844a-428f-b295-b7500cdeb3d0","Type":"ContainerStarted","Data":"ac3e3ae9a1bee138edc2aa56bef1a6d733627dcbbb30d7a0c9e2a471e0e30e9f"}
Oct 31 00:29:17.917530 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:17.917477    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podStartSLOduration=1.917462583 podStartE2EDuration="1.917462583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:17.914757235 +0000 UTC m=+1590.931421966" watchObservedRunningTime="2026-10-31 00:29:17.917462583 +0000 UTC m=+1590.934127295"
Oct 31 00:29:18.243498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:18.243463    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:18.245398 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:18.245359    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:18.245534 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:18.245456    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:19.245849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:19.245810    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:19.246487 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:19.246453    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:20.249473 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:20.249438    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:20.250067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:20.250034    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:21.245356 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:21.245323    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:21.245650 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:21.245623    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:22.244806 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:22.244767    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:22.245446 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:22.245413    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:23.246851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:23.246815    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:23.247493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:23.247459    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:24.256404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:24.255444    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:24.256404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:24.255500    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:25.250401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:25.247845    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:25.250401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:25.247909    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:26.243848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:26.243813    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body=
Oct 31 00:29:26.244480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:26.244438    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused"
Oct 31 00:29:27.720070 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:27.720038    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:27.721442 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:27.721343    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:27.724667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:27.724647    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:29:43.610811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.610768    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"]
Oct 31 00:29:43.615112 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.615088    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.669352 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.669299    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-835921-serving-cert\""
Oct 31 00:29:43.714435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.714368    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6f2c\" (UniqueName: \"kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.714649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.714627    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.714737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.714661    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.714848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.714821    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.722041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.722019    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"]
Oct 31 00:29:43.816334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.816300    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.816511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.816449    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6f2c\" (UniqueName: \"kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.816608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.816592    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.816651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.816623    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.816772 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:29:43.816750    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-835921-serving-cert: secret "maas-api-e2e-api-a-835921-serving-cert" not found
Oct 31 00:29:43.816841 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:29:43.816828    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls podName:d7d6d0ef-9171-403e-b17e-d9b21148241e nodeName:}" failed. No retries permitted until 2026-10-31 00:29:44.316807522 +0000 UTC m=+1617.333472225 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls") pod "maas-api-e2e-api-a-835921-668f458884-h4r72" (UID: "d7d6d0ef-9171-403e-b17e-d9b21148241e") : secret "maas-api-e2e-api-a-835921-serving-cert" not found
Oct 31 00:29:43.816974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.816954    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.819293 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.819271    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:43.878088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:43.878028    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6f2c\" (UniqueName: \"kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:44.331221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:44.331187    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:44.333875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:44.333850    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\") pod \"maas-api-e2e-api-a-835921-668f458884-h4r72\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:44.529238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:44.529207    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:44.758112 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:29:44.758075    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:29:44.910284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:44.910254    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"]
Oct 31 00:29:44.911772 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:29:44.911748    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d6d0ef_9171_403e_b17e_d9b21148241e.slice/crio-d7ffa4fe2b8bb53d170b1ee9cc46c665f7c0de27baf0ce435604c80fd4f797b0 WatchSource:0}: Error finding container d7ffa4fe2b8bb53d170b1ee9cc46c665f7c0de27baf0ce435604c80fd4f797b0: Status 404 returned error can't find the container with id d7ffa4fe2b8bb53d170b1ee9cc46c665f7c0de27baf0ce435604c80fd4f797b0
Oct 31 00:29:45.233696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:45.233664    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72" event={"ID":"d7d6d0ef-9171-403e-b17e-d9b21148241e","Type":"ContainerStarted","Data":"d7ffa4fe2b8bb53d170b1ee9cc46c665f7c0de27baf0ce435604c80fd4f797b0"}
Oct 31 00:29:46.242452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:46.242335    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72" event={"ID":"d7d6d0ef-9171-403e-b17e-d9b21148241e","Type":"ContainerStarted","Data":"f8e82ff022d12196a3da03db7d5ed3dc9f5be5d13b6aa93d9cb42546a536d303"}
Oct 31 00:29:46.242824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:46.242572    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:29:46.365592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:46.365538    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72" podStartSLOduration=2.337884465 podStartE2EDuration="3.365523909s" totalImagesPullingTime="1.027639444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:46.364161492 +0000 UTC m=+1619.380826205" watchObservedRunningTime="2026-10-31 00:29:46.365523909 +0000 UTC m=+1619.382188622"
Oct 31 00:29:52.266407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:29:52.265729    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:30:14.110116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.109907    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"]
Oct 31 00:30:14.113618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.113591    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.167423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.167373    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-f7c92a85-openshift-default-dockercfg-rt7pg\""
Oct 31 00:30:14.219926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.219898    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"]
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262491    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262594    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262628    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262711    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262741    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262775    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262830    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.262974    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.273406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.263020    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngnks\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.371142 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.371059    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.372195 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.372071    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.372412 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.372371    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngnks\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.372964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.372943    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.373426 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.373372    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.373944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.373926    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.374452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.374433    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.375152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.375133    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.375635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.375619    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.376467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.375570    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.377344 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.375058    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.377487 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.374305    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.377577 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.376653    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.377730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.377285    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.395566 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.395481    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.399988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.399959    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.464423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.450347    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.464423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.453203    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngnks\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-db92l\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.729722 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:14.729694    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:14.760951 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:30:14.760914    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:30:15.169521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.169499    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"]
Oct 31 00:30:15.171688 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:30:15.171664    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a696228_3344_41d5_8f66_6e0f12e86723.slice/crio-13ec9cc36c919931efee5445f4b6b03e327460ce37d39eef0e078a29b9a50b17 WatchSource:0}: Error finding container 13ec9cc36c919931efee5445f4b6b03e327460ce37d39eef0e078a29b9a50b17: Status 404 returned error can't find the container with id 13ec9cc36c919931efee5445f4b6b03e327460ce37d39eef0e078a29b9a50b17
Oct 31 00:30:15.174187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.174152    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:15.174275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.174221    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:15.174275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.174251    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:15.560348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.560262    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" event={"ID":"4a696228-3344-41d5-8f66-6e0f12e86723","Type":"ContainerStarted","Data":"9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e"}
Oct 31 00:30:15.560348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.560346    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" event={"ID":"4a696228-3344-41d5-8f66-6e0f12e86723","Type":"ContainerStarted","Data":"13ec9cc36c919931efee5445f4b6b03e327460ce37d39eef0e078a29b9a50b17"}
Oct 31 00:30:15.683781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.683713    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podStartSLOduration=1.683692755 podStartE2EDuration="1.683692755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:15.680643002 +0000 UTC m=+1648.697307753" watchObservedRunningTime="2026-10-31 00:30:15.683692755 +0000 UTC m=+1648.700357467"
Oct 31 00:30:15.730316 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:15.730283    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:16.731277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:16.731245    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:30:16.731812 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:16.731305    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 00:30:17.730221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:17.730189    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:30:17.730547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:17.730519    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 00:30:18.730116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:18.730084    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:30:18.730523 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:18.730137    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 00:30:19.730133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:19.730104    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:30:19.730547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:19.730154    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 00:30:20.730896 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:20.730858    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body=
Oct 31 00:30:20.731298 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:20.730921    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused"
Oct 31 00:30:21.734314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:21.734284    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:21.734771 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:21.734748    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:21.735435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:21.735408    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:30:32.522433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.522316    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"]
Oct 31 00:30:32.528014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.527987    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.569783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.569747    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.569936 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.569799    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570000 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.569980    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.570079    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.570188    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.570250    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.570296    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570399 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.570343    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.570501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.570483    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.645607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.645574    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"]
Oct 31 00:30:32.671510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671469    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.671730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671597    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.671730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671633    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.671730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671676    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.671730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671705    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671765    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671891    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.671934    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.672019    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.672349    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.672524    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.672849    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.673787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.673080    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.674190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.673909    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.675278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.675253    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.682095 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.682067    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.736040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.736011    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.737120 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.737092    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token\") pod \"e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") " pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:32.842615 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:32.842534    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:33.377077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.377054    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"]
Oct 31 00:30:33.379808 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:30:33.379781    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2697501d_54c8_4889_9d8c_3af0f7258922.slice/crio-6e00d137e7c13a13af70ff4a9aecf3c086be51bcaee165d90894f9996cf9d06c WatchSource:0}: Error finding container 6e00d137e7c13a13af70ff4a9aecf3c086be51bcaee165d90894f9996cf9d06c: Status 404 returned error can't find the container with id 6e00d137e7c13a13af70ff4a9aecf3c086be51bcaee165d90894f9996cf9d06c
Oct 31 00:30:33.383029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.382996    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:33.383106 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.383060    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:33.383106 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.383088    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:33.687436 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.687365    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" event={"ID":"2697501d-54c8-4889-9d8c-3af0f7258922","Type":"ContainerStarted","Data":"695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0"}
Oct 31 00:30:33.687907 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.687471    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" event={"ID":"2697501d-54c8-4889-9d8c-3af0f7258922","Type":"ContainerStarted","Data":"6e00d137e7c13a13af70ff4a9aecf3c086be51bcaee165d90894f9996cf9d06c"}
Oct 31 00:30:33.809279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.809222    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podStartSLOduration=1.809208371 podStartE2EDuration="1.809208371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:33.806975316 +0000 UTC m=+1666.823640027" watchObservedRunningTime="2026-10-31 00:30:33.809208371 +0000 UTC m=+1666.825873083"
Oct 31 00:30:33.843191 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:33.843160    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:34.857782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:34.857749    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:30:34.858178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:34.857803    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 00:30:35.854407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:35.851708    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:30:35.854407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:35.851765    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 00:30:36.849406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:36.846725    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:30:36.849406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:36.846793    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 00:30:37.800417 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.785291    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"]
Oct 31 00:30:37.838408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.835666    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.856336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.851556    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:30:37.856336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.851621    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920298    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920349    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zclz\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920398    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920465    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920551    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920681    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920708    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920786    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.920832    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:37.941406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.923268    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"]
Oct 31 00:30:37.975403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:37.945000    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-2a59a7-openshift-default-dockercfg-rtd7g\""
Oct 31 00:30:38.108185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.108045    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.275606    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.275687    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.275779    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.275820    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.276008    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.276034    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zclz\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.276063    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.276136    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.276187    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.277008    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.277338    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.278513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.278141    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.333406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.290270    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.333406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.296823    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.354046 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.354013    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.395866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.395792    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zclz\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.417757 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.417678    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token\") pod \"e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") " pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.684957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.684928    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:38.850460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.847636    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:30:38.850460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:38.847731    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 00:30:39.076171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.072421    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"]
Oct 31 00:30:39.078294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.078271    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.138129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.130975    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2ccdde-serving-cert\""
Oct 31 00:30:39.176047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.176007    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"]
Oct 31 00:30:39.179833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.179802    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.179968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.179897    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r4st\" (UniqueName: \"kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.180029 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.180010    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.180097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.180033    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.199922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.199894    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"]
Oct 31 00:30:39.272340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.272300    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:39.272555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.272534    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:39.272689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.272673    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:30:39.281248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.281210    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.281415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.281371    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r4st\" (UniqueName: \"kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.281696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.281671    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.281779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.281716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.282093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.282069    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.282616 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:30:39.282548    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-serving-cert: secret "maas-api-e2e-api-b-2ccdde-serving-cert" not found
Oct 31 00:30:39.282616 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:30:39.282594    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls podName:3e03964e-6172-468c-a123-fea4cb43564a nodeName:}" failed. No retries permitted until 2026-10-31 00:30:39.782579536 +0000 UTC m=+1672.799244240 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls") pod "maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" (UID: "3e03964e-6172-468c-a123-fea4cb43564a") : secret "maas-api-e2e-api-b-2ccdde-serving-cert" not found
Oct 31 00:30:39.285150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.285125    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.381409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.370525    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r4st\" (UniqueName: \"kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.796363 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.796098    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.799168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.799142    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\") pod \"maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:39.814829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.814785    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" event={"ID":"f4564be4-23f3-4588-a910-7f3f8d1ec13e","Type":"ContainerStarted","Data":"5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"}
Oct 31 00:30:39.814958 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.814847    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" event={"ID":"f4564be4-23f3-4588-a910-7f3f8d1ec13e","Type":"ContainerStarted","Data":"61370d95557fff179972f400be46ec49e6b629e898f61fae499ccb2b4f01423c"}
Oct 31 00:30:39.843639 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.843605    4478 patch_prober.go:29] interesting pod/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body=
Oct 31 00:30:39.843822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.843671    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused"
Oct 31 00:30:39.938372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.938302    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podStartSLOduration=2.938285309 podStartE2EDuration="2.938285309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:39.936782047 +0000 UTC m=+1672.953446780" watchObservedRunningTime="2026-10-31 00:30:39.938285309 +0000 UTC m=+1672.954950027"
Oct 31 00:30:39.993634 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:39.993602    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:40.454515 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:40.454489    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"]
Oct 31 00:30:40.457373 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:30:40.457340    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e03964e_6172_468c_a123_fea4cb43564a.slice/crio-cb9cc3d2c5a4da439c73c49161612fb5d92c8a1a6e2e795623db81dc15ec9e3e WatchSource:0}: Error finding container cb9cc3d2c5a4da439c73c49161612fb5d92c8a1a6e2e795623db81dc15ec9e3e: Status 404 returned error can't find the container with id cb9cc3d2c5a4da439c73c49161612fb5d92c8a1a6e2e795623db81dc15ec9e3e
Oct 31 00:30:40.686070 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:40.686032    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:40.692478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:40.692454    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:40.831515 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:40.831413    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" event={"ID":"3e03964e-6172-468c-a123-fea4cb43564a","Type":"ContainerStarted","Data":"cb9cc3d2c5a4da439c73c49161612fb5d92c8a1a6e2e795623db81dc15ec9e3e"}
Oct 31 00:30:40.831909 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:40.831884    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:40.832919 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:40.832869    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:30:41.203890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.203860    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:41.207202 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.207172    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:41.208060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.208036    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:30:41.228719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.228693    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"]
Oct 31 00:30:41.476955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.463484    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"]
Oct 31 00:30:41.861260 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.861230    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" event={"ID":"3e03964e-6172-468c-a123-fea4cb43564a","Type":"ContainerStarted","Data":"bbb10fe67e874c95b0a073a14e8e87e6051e1f4babb184c7f04de4232fef47d9"}
Oct 31 00:30:41.861539 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:41.861524    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:44.850696 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:30:44.850600    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:30:46.229454 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:46.229416    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" containerName="payload-processing" containerID="cri-o://71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352" gracePeriod=25
Oct 31 00:30:46.478296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:46.478258    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerName="payload-pre-processing" containerID="cri-o://211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9" gracePeriod=25
Oct 31 00:30:48.894192 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:48.894160    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:30:49.014763 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:49.014696    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" podStartSLOduration=9.031850536 podStartE2EDuration="10.014677545s" totalImagesPullingTime="982.827009ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:41.9918537 +0000 UTC m=+1675.008518422" watchObservedRunningTime="2026-10-31 00:30:49.014677545 +0000 UTC m=+1682.031342261"
Oct 31 00:30:51.225847 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.225807    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"]
Oct 31 00:30:51.287532 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.284012    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"]
Oct 31 00:30:51.287532 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.284137    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.366104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.366071    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2a59a7\""
Oct 31 00:30:51.460111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.456754    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2a59a7-dockercfg-jr8qk\""
Oct 31 00:30:51.474237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.470189    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx6r7\" (UniqueName: \"kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7\") pod \"payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.474237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.470243    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.571070 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.570978    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.571266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.571249    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx6r7\" (UniqueName: \"kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7\") pod \"payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.571675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.571657    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.622115 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.622087    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:51.622250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.622091    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:30:51.631870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.631847    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx6r7\" (UniqueName: \"kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7\") pod \"payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") " pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:51.705422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:51.705367    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:52.086971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:52.086939    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"]
Oct 31 00:30:52.087508 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:30:52.087480    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod942a7b16_b002_48e1_88b3_d08d6c0ea13a.slice/crio-23eb08f3c9e4d1b7b9f1e38dfb4b856351c5c90bf65a2ee115a710acb4f004d5 WatchSource:0}: Error finding container 23eb08f3c9e4d1b7b9f1e38dfb4b856351c5c90bf65a2ee115a710acb4f004d5: Status 404 returned error can't find the container with id 23eb08f3c9e4d1b7b9f1e38dfb4b856351c5c90bf65a2ee115a710acb4f004d5
Oct 31 00:30:52.177712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:52.177568    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz" event={"ID":"942a7b16-b002-48e1-88b3-d08d6c0ea13a","Type":"ContainerStarted","Data":"d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709"}
Oct 31 00:30:52.177712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:52.177637    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz" event={"ID":"942a7b16-b002-48e1-88b3-d08d6c0ea13a","Type":"ContainerStarted","Data":"23eb08f3c9e4d1b7b9f1e38dfb4b856351c5c90bf65a2ee115a710acb4f004d5"}
Oct 31 00:30:52.177712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:52.177666    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:30:52.293729 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:52.293681    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz" podStartSLOduration=1.293667661 podStartE2EDuration="1.293667661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:52.292773623 +0000 UTC m=+1685.309438353" watchObservedRunningTime="2026-10-31 00:30:52.293667661 +0000 UTC m=+1685.310332406"
Oct 31 00:30:58.669420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.663442    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"]
Oct 31 00:30:58.697406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.682932    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:58.706296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.706259    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"]
Oct 31 00:30:58.721059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.720966    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7flb\" (UniqueName: \"kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb\") pod \"payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:58.721196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.721131    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:58.785414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.785360    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bba5ed-dockercfg-qdsm7\""
Oct 31 00:30:58.785567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.785470    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bba5ed\""
Oct 31 00:30:58.822067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.822034    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7flb\" (UniqueName: \"kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb\") pod \"payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:58.822239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.822162    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:58.822873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.822852    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:58.882977 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:58.882952    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7flb\" (UniqueName: \"kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb\") pod \"payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:59.022590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:59.022561    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:30:59.458332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:30:59.458299    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"]
Oct 31 00:31:00.260944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:00.260904    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz" event={"ID":"6f5ea501-6174-4d59-9841-62e7f4084a92","Type":"ContainerStarted","Data":"b22119e5e7c75d2e54f624d530a1d03747562b5d10ec158a9dbd3594174f24ef"}
Oct 31 00:31:00.261327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:00.260966    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz" event={"ID":"6f5ea501-6174-4d59-9841-62e7f4084a92","Type":"ContainerStarted","Data":"21f203044c0e361eaa75e01f2e1c727f5144a828a0032d62fe326016d7f36a60"}
Oct 31 00:31:00.261327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:00.261016    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:31:01.621624 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:01.621582    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:01.622027 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:01.621582    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:31:03.187072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:03.187046    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:31:03.309421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:03.309351    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz" podStartSLOduration=5.3093376039999995 podStartE2EDuration="5.309337604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:00.394728456 +0000 UTC m=+1693.411393186" watchObservedRunningTime="2026-10-31 00:31:03.309337604 +0000 UTC m=+1696.326002348"
Oct 31 00:31:06.583937 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.583910    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:31:06.635521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.635489    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd\" (UniqueName: \"kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd\") pod \"69da6e4c-5565-405d-92ea-75815f60ee05\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") "
Oct 31 00:31:06.635697 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.635548    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume\") pod \"69da6e4c-5565-405d-92ea-75815f60ee05\" (UID: \"69da6e4c-5565-405d-92ea-75815f60ee05\") "
Oct 31 00:31:06.635898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.635873    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume" pod "69da6e4c-5565-405d-92ea-75815f60ee05" (UID: "69da6e4c-5565-405d-92ea-75815f60ee05"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:06.636253 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.636235    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69da6e4c-5565-405d-92ea-75815f60ee05-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:06.637861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.637826    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd" pod "69da6e4c-5565-405d-92ea-75815f60ee05" (UID: "69da6e4c-5565-405d-92ea-75815f60ee05"). InnerVolumeSpecName "kube-api-access-gq6pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:06.737426 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.737325    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq6pd\" (UniqueName: \"kubernetes.io/projected/69da6e4c-5565-405d-92ea-75815f60ee05-kube-api-access-gq6pd\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:06.756502 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.756479    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:31:06.838483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.838448    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp\" (UniqueName: \"kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp\") pod \"fbd12343-79c6-48e7-9e9c-fc7abe167962\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") "
Oct 31 00:31:06.838679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.838538    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume\") pod \"fbd12343-79c6-48e7-9e9c-fc7abe167962\" (UID: \"fbd12343-79c6-48e7-9e9c-fc7abe167962\") "
Oct 31 00:31:06.838990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.838957    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume" pod "fbd12343-79c6-48e7-9e9c-fc7abe167962" (UID: "fbd12343-79c6-48e7-9e9c-fc7abe167962"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:06.839270 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.839240    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbd12343-79c6-48e7-9e9c-fc7abe167962-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:06.840932 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.840909    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp" pod "fbd12343-79c6-48e7-9e9c-fc7abe167962" (UID: "fbd12343-79c6-48e7-9e9c-fc7abe167962"). InnerVolumeSpecName "kube-api-access-njwnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:06.940316 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:06.940281    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njwnp\" (UniqueName: \"kubernetes.io/projected/fbd12343-79c6-48e7-9e9c-fc7abe167962-kube-api-access-njwnp\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:07.314284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.314252    4478 generic.go:356] "Generic (PLEG): container finished" podID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerID="211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9" exitCode=0
Oct 31 00:31:07.314469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.314296    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"
Oct 31 00:31:07.314469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.314340    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" event={"ID":"fbd12343-79c6-48e7-9e9c-fc7abe167962","Type":"ContainerDied","Data":"211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9"}
Oct 31 00:31:07.314469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.314424    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx" event={"ID":"fbd12343-79c6-48e7-9e9c-fc7abe167962","Type":"ContainerDied","Data":"06737ff6f023d9c39534a03653a8d30569fb1b5a5eab8f3328dbceec7ed3d5a1"}
Oct 31 00:31:07.314469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.314455    4478 scope.go:122] "RemoveContainer" containerID="211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9"
Oct 31 00:31:07.315897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.315869    4478 generic.go:356] "Generic (PLEG): container finished" podID="69da6e4c-5565-405d-92ea-75815f60ee05" containerID="71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352" exitCode=0
Oct 31 00:31:07.316002 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.315959    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" event={"ID":"69da6e4c-5565-405d-92ea-75815f60ee05","Type":"ContainerDied","Data":"71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352"}
Oct 31 00:31:07.316073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.316017    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8" event={"ID":"69da6e4c-5565-405d-92ea-75815f60ee05","Type":"ContainerDied","Data":"946107ee0351488cab118c28a50c9c5cd0968cb69a6dcbb74dbcc9ea48dda286"}
Oct 31 00:31:07.316124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.316082    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"
Oct 31 00:31:07.324361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.324344    4478 scope.go:122] "RemoveContainer" containerID="211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9"
Oct 31 00:31:07.324654 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:07.324636    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9\": container with ID starting with 211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9 not found: ID does not exist" containerID="211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9"
Oct 31 00:31:07.324736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.324664    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9"} err="failed to get container status \"211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9\": rpc error: code = NotFound desc = could not find container \"211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9\": container with ID starting with 211c71b54c987f2d23f46adfb5096df09beea4ae65bfe2205cc3bb26a43f04a9 not found: ID does not exist"
Oct 31 00:31:07.324736 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.324687    4478 scope.go:122] "RemoveContainer" containerID="71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352"
Oct 31 00:31:07.333180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.333148    4478 scope.go:122] "RemoveContainer" containerID="71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352"
Oct 31 00:31:07.333475 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:07.333458    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352\": container with ID starting with 71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352 not found: ID does not exist" containerID="71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352"
Oct 31 00:31:07.333533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.333481    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352"} err="failed to get container status \"71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352\": rpc error: code = NotFound desc = could not find container \"71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352\": container with ID starting with 71b6f0e98f32c173293cf3ff21f8bee4925b54ec566a9496dc2cf3eb30dfd352 not found: ID does not exist"
Oct 31 00:31:07.478842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.478805    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"]
Oct 31 00:31:07.529241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.529211    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-f7c92a85-7b65f87cdc-l6sgx"]
Oct 31 00:31:07.595196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.595163    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"]
Oct 31 00:31:07.602496 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:07.602468    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-7kjb8"]
Oct 31 00:31:09.117859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:09.117827    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" path="/var/lib/kubelet/pods/69da6e4c-5565-405d-92ea-75815f60ee05/volumes"
Oct 31 00:31:09.118258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:09.118243    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" path="/var/lib/kubelet/pods/fbd12343-79c6-48e7-9e9c-fc7abe167962/volumes"
Oct 31 00:31:11.269836 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:11.269812    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:31:14.926428 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:14.926365    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:31:23.853302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.853268    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"]
Oct 31 00:31:23.854480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.853888    4478 cpu_manager.go:414] "containerMap: removing container" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" containerName="payload-processing"
Oct 31 00:31:23.854480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.853901    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" containerName="payload-processing"
Oct 31 00:31:23.854480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.853935    4478 cpu_manager.go:414] "containerMap: removing container" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerName="payload-pre-processing"
Oct 31 00:31:23.854480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.853942    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerName="payload-pre-processing"
Oct 31 00:31:23.854480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.854032    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbd12343-79c6-48e7-9e9c-fc7abe167962" containerName="payload-pre-processing"
Oct 31 00:31:23.854480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.854044    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="69da6e4c-5565-405d-92ea-75815f60ee05" containerName="payload-processing"
Oct 31 00:31:23.858843 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.858821    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:23.898595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.898569    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:23.898708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.898605    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:23.898708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.898696    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qws9s\" (UniqueName: \"kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:23.898874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.898848    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:23.961866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.961833    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3e7872-serving-cert\""
Oct 31 00:31:23.965326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:23.965303    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"]
Oct 31 00:31:24.003684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.003646    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.003916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.003897    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.004091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.004072    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qws9s\" (UniqueName: \"kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.004272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.004256    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.005401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.005359    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.005791 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:24.003823    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-serving-cert: secret "maas-api-e2e-api-a-3e7872-serving-cert" not found
Oct 31 00:31:24.006043 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:24.006027    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls podName:03ad984f-11e3-40c8-be50-a874b1b755e6 nodeName:}" failed. No retries permitted until 2026-10-31 00:31:24.506004558 +0000 UTC m=+1717.522669266 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls") pod "maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" (UID: "03ad984f-11e3-40c8-be50-a874b1b755e6") : secret "maas-api-e2e-api-a-3e7872-serving-cert" not found
Oct 31 00:31:24.008496 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.008472    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.067877 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.067838    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qws9s\" (UniqueName: \"kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.567531 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.567497    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.570047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.570017    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\") pod \"maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:24.775241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:24.775200    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:25.162036 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:31:25.162005    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ad984f_11e3_40c8_be50_a874b1b755e6.slice/crio-f3b50538bd63637203756ab8c65319134a2596d040344105d834ac79757b3ad1 WatchSource:0}: Error finding container f3b50538bd63637203756ab8c65319134a2596d040344105d834ac79757b3ad1: Status 404 returned error can't find the container with id f3b50538bd63637203756ab8c65319134a2596d040344105d834ac79757b3ad1
Oct 31 00:31:25.162489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:25.162070    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"]
Oct 31 00:31:25.444460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:25.444428    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" event={"ID":"03ad984f-11e3-40c8-be50-a874b1b755e6","Type":"ContainerStarted","Data":"f3b50538bd63637203756ab8c65319134a2596d040344105d834ac79757b3ad1"}
Oct 31 00:31:26.454340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:26.454309    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" event={"ID":"03ad984f-11e3-40c8-be50-a874b1b755e6","Type":"ContainerStarted","Data":"c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3"}
Oct 31 00:31:26.454891 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:26.454541    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:26.574323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:26.574255    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" podStartSLOduration=2.714580586 podStartE2EDuration="3.574238684s" totalImagesPullingTime="859.658098ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:26.574215961 +0000 UTC m=+1719.590880689" watchObservedRunningTime="2026-10-31 00:31:26.574238684 +0000 UTC m=+1719.590903398"
Oct 31 00:31:32.466527 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:32.466493    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:31:33.593538 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.593507    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"]
Oct 31 00:31:33.598792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.598765    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.702397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.702347    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-2a59a7-serving-cert\""
Oct 31 00:31:33.706004 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.705976    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"]
Oct 31 00:31:33.708101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.708074    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.708214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.708200    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97vlf\" (UniqueName: \"kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.708279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.708227    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.708350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.708339    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.809239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.809207    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97vlf\" (UniqueName: \"kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.809445 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.809253    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.809445 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.809331    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.809583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.809510    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.809669 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:33.809636    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-serving-cert: secret "maas-api-e2e-api-b-2a59a7-serving-cert" not found
Oct 31 00:31:33.809742 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:33.809724    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls podName:2484b7fe-bf88-477e-8b56-4a82a8b3b406 nodeName:}" failed. No retries permitted until 2026-10-31 00:31:34.309702417 +0000 UTC m=+1727.326367107 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls") pod "maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" (UID: "2484b7fe-bf88-477e-8b56-4a82a8b3b406") : secret "maas-api-e2e-api-b-2a59a7-serving-cert" not found
Oct 31 00:31:33.810071 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.810051    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.812037 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.812017    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:33.920451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:33.914211    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97vlf\" (UniqueName: \"kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:34.336554 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:34.336521    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:34.339252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:34.339217    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\") pod \"maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:34.511978 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:34.511948    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:34.902958 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:34.902927    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"]
Oct 31 00:31:34.903477 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:31:34.903452    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2484b7fe_bf88_477e_8b56_4a82a8b3b406.slice/crio-a16849a3671f934e66a8ceb51f558f7d0a921e09f3731e41e44ae7cf59d66417 WatchSource:0}: Error finding container a16849a3671f934e66a8ceb51f558f7d0a921e09f3731e41e44ae7cf59d66417: Status 404 returned error can't find the container with id a16849a3671f934e66a8ceb51f558f7d0a921e09f3731e41e44ae7cf59d66417
Oct 31 00:31:35.524475 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:35.524439    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" event={"ID":"2484b7fe-bf88-477e-8b56-4a82a8b3b406","Type":"ContainerStarted","Data":"a16849a3671f934e66a8ceb51f558f7d0a921e09f3731e41e44ae7cf59d66417"}
Oct 31 00:31:36.533876 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:36.533832    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" event={"ID":"2484b7fe-bf88-477e-8b56-4a82a8b3b406","Type":"ContainerStarted","Data":"833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9"}
Oct 31 00:31:36.534299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:36.534046    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:36.652594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:36.652526    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" podStartSLOduration=2.80953308 podStartE2EDuration="3.652507701s" totalImagesPullingTime="842.974621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:36.650056764 +0000 UTC m=+1729.666721492" watchObservedRunningTime="2026-10-31 00:31:36.652507701 +0000 UTC m=+1729.669172414"
Oct 31 00:31:42.545329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:42.545297    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:31:44.759019 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:44.758976    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:31:49.006795 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.006756    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"]
Oct 31 00:31:49.012264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.012240    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.051455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.051421    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.051592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.051538    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.051728 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.051706    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.051798 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.051786    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwv5n\" (UniqueName: \"kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.115548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.115519    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bba5ed-serving-cert\""
Oct 31 00:31:49.135251 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.135219    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"]
Oct 31 00:31:49.152829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.152785    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.153895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.153871    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.154178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.154158    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.154361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.154342    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwv5n\" (UniqueName: \"kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.154870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.153761    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.156532 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:49.156514    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-serving-cert: secret "maas-api-e2e-ipp-a-bba5ed-serving-cert" not found
Oct 31 00:31:49.156692 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:49.156678    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls podName:ec54d20e-c588-4c99-8a58-1157e796616a nodeName:}" failed. No retries permitted until 2026-10-31 00:31:49.656658352 +0000 UTC m=+1742.673323047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls") pod "maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" (UID: "ec54d20e-c588-4c99-8a58-1157e796616a") : secret "maas-api-e2e-ipp-a-bba5ed-serving-cert" not found
Oct 31 00:31:49.162305 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.162259    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.215443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.215413    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwv5n\" (UniqueName: \"kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.697330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.697296    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.699896 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.699871    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:49.925876 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:49.925840    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:50.314722 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:50.314694    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"]
Oct 31 00:31:50.315838 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:31:50.315810    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec54d20e_c588_4c99_8a58_1157e796616a.slice/crio-0073eaa758678c66d7cb61ed012a2ad4ec7514a782a10c7535ab623956309882 WatchSource:0}: Error finding container 0073eaa758678c66d7cb61ed012a2ad4ec7514a782a10c7535ab623956309882: Status 404 returned error can't find the container with id 0073eaa758678c66d7cb61ed012a2ad4ec7514a782a10c7535ab623956309882
Oct 31 00:31:50.621886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:50.621803    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" event={"ID":"ec54d20e-c588-4c99-8a58-1157e796616a","Type":"ContainerStarted","Data":"0073eaa758678c66d7cb61ed012a2ad4ec7514a782a10c7535ab623956309882"}
Oct 31 00:31:51.642608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:51.636991    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" event={"ID":"ec54d20e-c588-4c99-8a58-1157e796616a","Type":"ContainerStarted","Data":"91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23"}
Oct 31 00:31:51.642608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:51.637971    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:51.760188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:51.760123    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" podStartSLOduration=2.772619991 podStartE2EDuration="3.760108786s" totalImagesPullingTime="987.488795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:31:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:31:51.75752215 +0000 UTC m=+1744.774186863" watchObservedRunningTime="2026-10-31 00:31:51.760108786 +0000 UTC m=+1744.776773498"
Oct 31 00:31:52.454033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:52.454003    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"]
Oct 31 00:31:52.566993 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:52.566957    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"]
Oct 31 00:31:57.454366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.454292    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk" podUID="d47ce4eb-36e7-47f1-ae35-ef6188200464" containerName="payload-processing" containerID="cri-o://eb4cc77384e2ed5dc8ad7ac6858e2ec745224fbcaeb47b3727820c60bff5ba6b" gracePeriod=25
Oct 31 00:31:57.568478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.568434    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz" podUID="0c1b72a5-b65d-495c-9a69-5e341c8c8973" containerName="payload-pre-processing" containerID="cri-o://faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675" gracePeriod=25
Oct 31 00:31:57.687271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.687244    4478 generic.go:356] "Generic (PLEG): container finished" podID="d47ce4eb-36e7-47f1-ae35-ef6188200464" containerID="eb4cc77384e2ed5dc8ad7ac6858e2ec745224fbcaeb47b3727820c60bff5ba6b" exitCode=0
Oct 31 00:31:57.687371 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.687342    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk" event={"ID":"d47ce4eb-36e7-47f1-ae35-ef6188200464","Type":"ContainerDied","Data":"eb4cc77384e2ed5dc8ad7ac6858e2ec745224fbcaeb47b3727820c60bff5ba6b"}
Oct 31 00:31:57.743035 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.743007    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:31:57.866613 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.866586    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:31:57.914720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.914693    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l\" (UniqueName: \"kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l\") pod \"d47ce4eb-36e7-47f1-ae35-ef6188200464\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") "
Oct 31 00:31:57.914952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.914933    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume\") pod \"d47ce4eb-36e7-47f1-ae35-ef6188200464\" (UID: \"d47ce4eb-36e7-47f1-ae35-ef6188200464\") "
Oct 31 00:31:57.915313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.915285    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume" pod "d47ce4eb-36e7-47f1-ae35-ef6188200464" (UID: "d47ce4eb-36e7-47f1-ae35-ef6188200464"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:57.915627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.915503    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d47ce4eb-36e7-47f1-ae35-ef6188200464-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:57.917034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:57.917013    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l" pod "d47ce4eb-36e7-47f1-ae35-ef6188200464" (UID: "d47ce4eb-36e7-47f1-ae35-ef6188200464"). InnerVolumeSpecName "kube-api-access-5fz2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:58.016915 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.016837    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume\") pod \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") "
Oct 31 00:31:58.016915 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.016901    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6\" (UniqueName: \"kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6\") pod \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\" (UID: \"0c1b72a5-b65d-495c-9a69-5e341c8c8973\") "
Oct 31 00:31:58.017301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.017272    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume" pod "0c1b72a5-b65d-495c-9a69-5e341c8c8973" (UID: "0c1b72a5-b65d-495c-9a69-5e341c8c8973"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:31:58.017586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.017568    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c1b72a5-b65d-495c-9a69-5e341c8c8973-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.017647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.017593    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fz2l\" (UniqueName: \"kubernetes.io/projected/d47ce4eb-36e7-47f1-ae35-ef6188200464-kube-api-access-5fz2l\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.019474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.019454    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6" pod "0c1b72a5-b65d-495c-9a69-5e341c8c8973" (UID: "0c1b72a5-b65d-495c-9a69-5e341c8c8973"). InnerVolumeSpecName "kube-api-access-jtnf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:31:58.118855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.118819    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtnf6\" (UniqueName: \"kubernetes.io/projected/0c1b72a5-b65d-495c-9a69-5e341c8c8973-kube-api-access-jtnf6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:31:58.656443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.656414    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:31:58.695995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.695965    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"
Oct 31 00:31:58.696178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.696003    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk" event={"ID":"d47ce4eb-36e7-47f1-ae35-ef6188200464","Type":"ContainerDied","Data":"c57f75e07ef5c429b452bf747f136608ec828e376352346feac99e2a6fad5ef8"}
Oct 31 00:31:58.696178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.696058    4478 scope.go:122] "RemoveContainer" containerID="eb4cc77384e2ed5dc8ad7ac6858e2ec745224fbcaeb47b3727820c60bff5ba6b"
Oct 31 00:31:58.697656 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.697634    4478 generic.go:356] "Generic (PLEG): container finished" podID="0c1b72a5-b65d-495c-9a69-5e341c8c8973" containerID="faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675" exitCode=0
Oct 31 00:31:58.697782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.697682    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"
Oct 31 00:31:58.697782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.697717    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz" event={"ID":"0c1b72a5-b65d-495c-9a69-5e341c8c8973","Type":"ContainerDied","Data":"faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675"}
Oct 31 00:31:58.697782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.697746    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz" event={"ID":"0c1b72a5-b65d-495c-9a69-5e341c8c8973","Type":"ContainerDied","Data":"62d498146e16bbc306d8ab7235dea456c886ef1509daaa1681ac9f0e30278d63"}
Oct 31 00:31:58.707449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.707432    4478 scope.go:122] "RemoveContainer" containerID="faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675"
Oct 31 00:31:58.716255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.716239    4478 scope.go:122] "RemoveContainer" containerID="faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675"
Oct 31 00:31:58.716608 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:31:58.716575    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675\": container with ID starting with faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675 not found: ID does not exist" containerID="faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675"
Oct 31 00:31:58.716662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.716616    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675"} err="failed to get container status \"faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675\": rpc error: code = NotFound desc = could not find container \"faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675\": container with ID starting with faa703b6d657f3beee0cb5fb5c1ccf0f640e5fae1876f88d05f9bf6ff0ac5675 not found: ID does not exist"
Oct 31 00:31:58.886837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.886754    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"]
Oct 31 00:31:58.895038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:58.895011    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-a4dc2e-6768b689f-4xwmz"]
Oct 31 00:31:59.002047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:59.002011    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"]
Oct 31 00:31:59.006246 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:59.006222    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a4dc2e-7c9cc6457-2dmdk"]
Oct 31 00:31:59.161130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:59.153982    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c1b72a5-b65d-495c-9a69-5e341c8c8973" path="/var/lib/kubelet/pods/0c1b72a5-b65d-495c-9a69-5e341c8c8973/volumes"
Oct 31 00:31:59.161130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:31:59.154673    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d47ce4eb-36e7-47f1-ae35-ef6188200464" path="/var/lib/kubelet/pods/d47ce4eb-36e7-47f1-ae35-ef6188200464/volumes"
Oct 31 00:32:03.153023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.152996    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"]
Oct 31 00:32:03.153469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.153184    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" podUID="7e8f1315-25e9-4b99-905f-72c5e925086d" containerName="maas-api" containerID="cri-o://5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791" gracePeriod=30
Oct 31 00:32:03.521588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.521559    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:32:03.717626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.715660    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs\") pod \"7e8f1315-25e9-4b99-905f-72c5e925086d\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") "
Oct 31 00:32:03.717626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.715847    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca\") pod \"7e8f1315-25e9-4b99-905f-72c5e925086d\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") "
Oct 31 00:32:03.717626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.715948    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542\" (UniqueName: \"kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542\") pod \"7e8f1315-25e9-4b99-905f-72c5e925086d\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") "
Oct 31 00:32:03.717626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.716101    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\") pod \"7e8f1315-25e9-4b99-905f-72c5e925086d\" (UID: \"7e8f1315-25e9-4b99-905f-72c5e925086d\") "
Oct 31 00:32:03.721035 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.718324    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca" pod "7e8f1315-25e9-4b99-905f-72c5e925086d" (UID: "7e8f1315-25e9-4b99-905f-72c5e925086d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:03.756848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.756165    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs" pod "7e8f1315-25e9-4b99-905f-72c5e925086d" (UID: "7e8f1315-25e9-4b99-905f-72c5e925086d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:03.756848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.756671    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542" pod "7e8f1315-25e9-4b99-905f-72c5e925086d" (UID: "7e8f1315-25e9-4b99-905f-72c5e925086d"). InnerVolumeSpecName "kube-api-access-82542". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:03.760486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.759645    4478 generic.go:356] "Generic (PLEG): container finished" podID="7e8f1315-25e9-4b99-905f-72c5e925086d" containerID="5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791" exitCode=0
Oct 31 00:32:03.760486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.759763    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" event={"ID":"7e8f1315-25e9-4b99-905f-72c5e925086d","Type":"ContainerDied","Data":"5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791"}
Oct 31 00:32:03.760486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.759813    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n" event={"ID":"7e8f1315-25e9-4b99-905f-72c5e925086d","Type":"ContainerDied","Data":"9e3b3406e85d1ab6a473f61c85ac7e3ee21ad5898317a1cc27d5de2f82bd7c4d"}
Oct 31 00:32:03.760486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.759843    4478 scope.go:122] "RemoveContainer" containerID="5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791"
Oct 31 00:32:03.760486 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.759982    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"
Oct 31 00:32:03.762586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.762537    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls" pod "7e8f1315-25e9-4b99-905f-72c5e925086d" (UID: "7e8f1315-25e9-4b99-905f-72c5e925086d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:32:03.800145 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.799847    4478 scope.go:122] "RemoveContainer" containerID="5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791"
Oct 31 00:32:03.802259 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:32:03.802168    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791\": container with ID starting with 5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791 not found: ID does not exist" containerID="5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791"
Oct 31 00:32:03.802259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.802213    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791"} err="failed to get container status \"5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791\": rpc error: code = NotFound desc = could not find container \"5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791\": container with ID starting with 5db9abe9c84bb1c5ddc55cd0a78f883531a58a3e9ab7ee8cde50138681e95791 not found: ID does not exist"
Oct 31 00:32:03.817920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.817887    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:03.818097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.817924    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e8f1315-25e9-4b99-905f-72c5e925086d-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:03.818097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.817942    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82542\" (UniqueName: \"kubernetes.io/projected/7e8f1315-25e9-4b99-905f-72c5e925086d-kube-api-access-82542\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:03.818097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:03.817956    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e8f1315-25e9-4b99-905f-72c5e925086d-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:04.203346 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:04.203309    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"]
Oct 31 00:32:04.251360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:04.251312    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-f7c92a85-546796bb4b-bdm4n"]
Oct 31 00:32:05.136416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:05.134864    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e8f1315-25e9-4b99-905f-72c5e925086d" path="/var/lib/kubelet/pods/7e8f1315-25e9-4b99-905f-72c5e925086d/volumes"
Oct 31 00:32:14.761341 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:32:14.761298    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:32:20.503331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.503295    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"]
Oct 31 00:32:20.504606 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504573    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d47ce4eb-36e7-47f1-ae35-ef6188200464" containerName="payload-processing"
Oct 31 00:32:20.504606 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504600    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d47ce4eb-36e7-47f1-ae35-ef6188200464" containerName="payload-processing"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504637    4478 cpu_manager.go:414] "containerMap: removing container" podUID="7e8f1315-25e9-4b99-905f-72c5e925086d" containerName="maas-api"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504646    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e8f1315-25e9-4b99-905f-72c5e925086d" containerName="maas-api"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504669    4478 cpu_manager.go:414] "containerMap: removing container" podUID="0c1b72a5-b65d-495c-9a69-5e341c8c8973" containerName="payload-pre-processing"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504676    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c1b72a5-b65d-495c-9a69-5e341c8c8973" containerName="payload-pre-processing"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504846    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d47ce4eb-36e7-47f1-ae35-ef6188200464" containerName="payload-processing"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504861    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e8f1315-25e9-4b99-905f-72c5e925086d" containerName="maas-api"
Oct 31 00:32:20.504924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.504872    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c1b72a5-b65d-495c-9a69-5e341c8c8973" containerName="payload-pre-processing"
Oct 31 00:32:20.509927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.509905    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:20.539625 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.539595    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"]
Oct 31 00:32:20.584433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.584314    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-f7c92a85\""
Oct 31 00:32:20.586813 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.586784    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-f7c92a85-dockercfg-v8zb4\""
Oct 31 00:32:20.731407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.724360    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-q75qn\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:20.731407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.724575    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzbm7\" (UniqueName: \"kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-q75qn\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:20.842453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.841689    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzbm7\" (UniqueName: \"kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-q75qn\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:20.842453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.841960    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-q75qn\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:20.846404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.843038    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-q75qn\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:20.906358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:20.906325    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzbm7\" (UniqueName: \"kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7\") pod \"payload-processing-e2e-mt-f7c92a85-67769656df-q75qn\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") " pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:21.126954 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:21.126874    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:21.515182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:21.515152    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"]
Oct 31 00:32:21.517003 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:32:21.516975    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048088a9_09d2_4410_8f47_f01884d7bfd5.slice/crio-2bc8850c771579facd736c50bfa5e4fd4cb39fb7f5b049fcbe997cae6f871d95 WatchSource:0}: Error finding container 2bc8850c771579facd736c50bfa5e4fd4cb39fb7f5b049fcbe997cae6f871d95: Status 404 returned error can't find the container with id 2bc8850c771579facd736c50bfa5e4fd4cb39fb7f5b049fcbe997cae6f871d95
Oct 31 00:32:21.887071 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:21.886970    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn" event={"ID":"048088a9-09d2-4410-8f47-f01884d7bfd5","Type":"ContainerStarted","Data":"034d7cc171beb54368182b09ef81b456a972f6c9ddcd6d43184acf9a684458e3"}
Oct 31 00:32:21.887071 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:21.887046    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:21.887297 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:21.887072    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn" event={"ID":"048088a9-09d2-4410-8f47-f01884d7bfd5","Type":"ContainerStarted","Data":"2bc8850c771579facd736c50bfa5e4fd4cb39fb7f5b049fcbe997cae6f871d95"}
Oct 31 00:32:22.018404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:22.017364    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn" podStartSLOduration=2.017343214 podStartE2EDuration="2.017343214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:22.011965526 +0000 UTC m=+1775.028630238" watchObservedRunningTime="2026-10-31 00:32:22.017343214 +0000 UTC m=+1775.034007927"
Oct 31 00:32:32.929194 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:32.929161    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:32:33.338072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:33.337984    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"]
Oct 31 00:32:33.338269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:33.338240    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy" containerID="cri-o://bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c" gracePeriod=30
Oct 31 00:32:38.701240 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.701217    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:32:38.794580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794546    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.794781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794622    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.794781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794665    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.794781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794696    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.794781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794739    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.795054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794802    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.795054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794841    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.795054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794898    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.795054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.794921    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo\") pod \"67adc0b3-30aa-4d12-99b7-860afd15ba60\" (UID: \"67adc0b3-30aa-4d12-99b7-860afd15ba60\") "
Oct 31 00:32:38.795263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.795052    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:32:38.795897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.795412    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:38.795897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.795602    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67adc0b3-30aa-4d12-99b7-860afd15ba60-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.795897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.795621    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.796155 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.795907    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:38.796155 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.796106    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:38.796268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.796185    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:38.797740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.797713    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:32:38.799034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.798997    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:32:38.799156 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.799098    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:38.802981 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.802954    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6" pod "67adc0b3-30aa-4d12-99b7-860afd15ba60" (UID: "67adc0b3-30aa-4d12-99b7-860afd15ba60"). InnerVolumeSpecName "kube-api-access-6g4k6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:32:38.897162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897126    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.897162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897166    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.897351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897181    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.897351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897198    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.897351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897211    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.897351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897225    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g4k6\" (UniqueName: \"kubernetes.io/projected/67adc0b3-30aa-4d12-99b7-860afd15ba60-kube-api-access-6g4k6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:38.897351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:38.897240    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67adc0b3-30aa-4d12-99b7-860afd15ba60-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:32:39.217691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.217656    4478 generic.go:356] "Generic (PLEG): container finished" podID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerID="bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c" exitCode=0
Oct 31 00:32:39.217879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.217697    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"
Oct 31 00:32:39.217879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.217715    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" event={"ID":"67adc0b3-30aa-4d12-99b7-860afd15ba60","Type":"ContainerDied","Data":"bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c"}
Oct 31 00:32:39.217879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.217769    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb" event={"ID":"67adc0b3-30aa-4d12-99b7-860afd15ba60","Type":"ContainerDied","Data":"f49edda89331cf5749eebc6da724242ccdd4eb9bbfc8ade3b72377a283205993"}
Oct 31 00:32:39.217879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.217796    4478 scope.go:122] "RemoveContainer" containerID="bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c"
Oct 31 00:32:39.227616 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.227596    4478 scope.go:122] "RemoveContainer" containerID="bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c"
Oct 31 00:32:39.227893 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:32:39.227872    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c\": container with ID starting with bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c not found: ID does not exist" containerID="bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c"
Oct 31 00:32:39.227960 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.227904    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c"} err="failed to get container status \"bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c\": rpc error: code = NotFound desc = could not find container \"bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c\": container with ID starting with bc06def53c77222c9986a182a60ae6d487bf334a94fdd2f952c83aef3408027c not found: ID does not exist"
Oct 31 00:32:39.383481 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.383375    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"]
Oct 31 00:32:39.462624 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:39.462582    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-zhfpb"]
Oct 31 00:32:41.121825 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.121793    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" path="/var/lib/kubelet/pods/67adc0b3-30aa-4d12-99b7-860afd15ba60/volumes"
Oct 31 00:32:41.484848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.484816    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"]
Oct 31 00:32:41.485818 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.485796    4478 cpu_manager.go:414] "containerMap: removing container" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy"
Oct 31 00:32:41.485912 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.485820    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy"
Oct 31 00:32:41.486059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.486046    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="67adc0b3-30aa-4d12-99b7-860afd15ba60" containerName="istio-proxy"
Oct 31 00:32:41.491805 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.491783    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.545693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.545647    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.545846 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.545752    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7x5n\" (UniqueName: \"kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n\") pod \"payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.546713 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.546692    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"]
Oct 31 00:32:41.590847 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.590801    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2cda64-dockercfg-xrnsb\""
Oct 31 00:32:41.591342 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.591306    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2cda64\""
Oct 31 00:32:41.627965 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.627910    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"]
Oct 31 00:32:41.868963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.656284    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.869128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.868977    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7x5n\" (UniqueName: \"kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n\") pod \"payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.869664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.657155    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.870725 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.870702    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"]
Oct 31 00:32:41.870977 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.870963    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:41.909647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.909611    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:32:41.909930 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.909893    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:32:41.930720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.930521    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7x5n\" (UniqueName: \"kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n\") pod \"payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:41.970541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.970516    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hczv\" (UniqueName: \"kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv\") pod \"payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:41.970710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:41.970655    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:42.072466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.072419    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hczv\" (UniqueName: \"kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv\") pod \"payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:42.072662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.072603    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:42.073547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.073513    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:42.114104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.114062    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:42.134219 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.134189    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hczv\" (UniqueName: \"kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv\") pod \"payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") " pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:42.204491 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.203324    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:42.503161 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.503133    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"]
Oct 31 00:32:42.505000 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:32:42.504965    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod332bac2a_f38e_4368_8534_b889aad36e42.slice/crio-0965ab34686546199d71f7c02bb868f8d5eb3bba56799ab9420b66c3edbc640e WatchSource:0}: Error finding container 0965ab34686546199d71f7c02bb868f8d5eb3bba56799ab9420b66c3edbc640e: Status 404 returned error can't find the container with id 0965ab34686546199d71f7c02bb868f8d5eb3bba56799ab9420b66c3edbc640e
Oct 31 00:32:42.607280 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:42.607251    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"]
Oct 31 00:32:42.609816 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:32:42.609784    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4ef832_f1b5_470b_828f_ebe2c31df5d9.slice/crio-70fa0516cdbc1d1ae4eeabd119f784f89e5b8836b157be2cd33d719001fd68d2 WatchSource:0}: Error finding container 70fa0516cdbc1d1ae4eeabd119f784f89e5b8836b157be2cd33d719001fd68d2: Status 404 returned error can't find the container with id 70fa0516cdbc1d1ae4eeabd119f784f89e5b8836b157be2cd33d719001fd68d2
Oct 31 00:32:43.278609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.278571    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g" event={"ID":"db4ef832-f1b5-470b-828f-ebe2c31df5d9","Type":"ContainerStarted","Data":"f55e07baf1eb445466ec18ef80d46cf2acb240a5fb68aba711c4f7029f10cefe"}
Oct 31 00:32:43.279086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.278639    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g" event={"ID":"db4ef832-f1b5-470b-828f-ebe2c31df5d9","Type":"ContainerStarted","Data":"70fa0516cdbc1d1ae4eeabd119f784f89e5b8836b157be2cd33d719001fd68d2"}
Oct 31 00:32:43.279086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.278667    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:43.280198 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.280173    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg" event={"ID":"332bac2a-f38e-4368-8534-b889aad36e42","Type":"ContainerStarted","Data":"b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502"}
Oct 31 00:32:43.280307 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.280214    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg" event={"ID":"332bac2a-f38e-4368-8534-b889aad36e42","Type":"ContainerStarted","Data":"0965ab34686546199d71f7c02bb868f8d5eb3bba56799ab9420b66c3edbc640e"}
Oct 31 00:32:43.280307 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.280275    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:43.395594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:43.395534    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g" podStartSLOduration=2.395516703 podStartE2EDuration="2.395516703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:43.393750719 +0000 UTC m=+1796.410415449" watchObservedRunningTime="2026-10-31 00:32:43.395516703 +0000 UTC m=+1796.412181416"
Oct 31 00:32:44.759276 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:32:44.759235    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:32:45.296285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.296212    4478 generic.go:356] "Generic (PLEG): container finished" podID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerID="5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37" exitCode=0
Oct 31 00:32:45.296497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.296287    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" event={"ID":"f4564be4-23f3-4588-a910-7f3f8d1ec13e","Type":"ContainerDied","Data":"5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"}
Oct 31 00:32:45.296752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.296730    4478 scope.go:122] "RemoveContainer" containerID="5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"
Oct 31 00:32:45.298506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.298478    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:32:45.298589 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.298575    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:32:45.298631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.298616    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:32:45.510990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:45.510924    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg" podStartSLOduration=4.510905398 podStartE2EDuration="4.510905398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:43.563811875 +0000 UTC m=+1796.580476621" watchObservedRunningTime="2026-10-31 00:32:45.510905398 +0000 UTC m=+1798.527570112"
Oct 31 00:32:46.305041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:46.305005    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" event={"ID":"f4564be4-23f3-4588-a910-7f3f8d1ec13e","Type":"ContainerStarted","Data":"18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde"}
Oct 31 00:32:46.685617 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:46.685579    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:32:46.686260 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:46.686234    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:32:46.686418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:46.686290    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 31 00:32:47.685681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:47.685639    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:32:47.686172 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:47.685696    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 31 00:32:48.686135 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:48.686102    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:32:48.686537 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:48.686169    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 31 00:32:49.686018 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:49.685978    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:32:49.686209 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:49.686061    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 31 00:32:50.687406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:50.687013    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:32:50.687406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:50.687079    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 31 00:32:51.685332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:51.685295    4478 patch_prober.go:29] interesting pod/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body=
Oct 31 00:32:51.685541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:51.685354    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused"
Oct 31 00:32:52.689712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:52.689682    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:32:52.690093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:52.689971    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:32:52.690613 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:52.690593    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:32:54.292764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:54.292735    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:32:54.293238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:54.293217    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:32:58.228698 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:32:58.228667    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"]
Oct 31 00:33:03.229123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.229069    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz" podUID="6f5ea501-6174-4d59-9841-62e7f4084a92" containerName="payload-processing" containerID="cri-o://b22119e5e7c75d2e54f624d530a1d03747562b5d10ec158a9dbd3594174f24ef" gracePeriod=25
Oct 31 00:33:03.453374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.453337    4478 generic.go:356] "Generic (PLEG): container finished" podID="6f5ea501-6174-4d59-9841-62e7f4084a92" containerID="b22119e5e7c75d2e54f624d530a1d03747562b5d10ec158a9dbd3594174f24ef" exitCode=0
Oct 31 00:33:03.453543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.453521    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz" event={"ID":"6f5ea501-6174-4d59-9841-62e7f4084a92","Type":"ContainerDied","Data":"b22119e5e7c75d2e54f624d530a1d03747562b5d10ec158a9dbd3594174f24ef"}
Oct 31 00:33:03.503688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.503656    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:33:03.544200 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.544170    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb\" (UniqueName: \"kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb\") pod \"6f5ea501-6174-4d59-9841-62e7f4084a92\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") "
Oct 31 00:33:03.544470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.544449    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume\") pod \"6f5ea501-6174-4d59-9841-62e7f4084a92\" (UID: \"6f5ea501-6174-4d59-9841-62e7f4084a92\") "
Oct 31 00:33:03.544790 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.544769    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume" pod "6f5ea501-6174-4d59-9841-62e7f4084a92" (UID: "6f5ea501-6174-4d59-9841-62e7f4084a92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:33:03.544996 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.544983    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f5ea501-6174-4d59-9841-62e7f4084a92-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:03.546452 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.546430    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb" pod "6f5ea501-6174-4d59-9841-62e7f4084a92" (UID: "6f5ea501-6174-4d59-9841-62e7f4084a92"). InnerVolumeSpecName "kube-api-access-z7flb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:03.647429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:03.647161    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7flb\" (UniqueName: \"kubernetes.io/projected/6f5ea501-6174-4d59-9841-62e7f4084a92-kube-api-access-z7flb\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:04.460978 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:04.460939    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz" event={"ID":"6f5ea501-6174-4d59-9841-62e7f4084a92","Type":"ContainerDied","Data":"21f203044c0e361eaa75e01f2e1c727f5144a828a0032d62fe326016d7f36a60"}
Oct 31 00:33:04.460978 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:04.460963    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"
Oct 31 00:33:04.461578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:04.461016    4478 scope.go:122] "RemoveContainer" containerID="b22119e5e7c75d2e54f624d530a1d03747562b5d10ec158a9dbd3594174f24ef"
Oct 31 00:33:04.660056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:04.655525    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"]
Oct 31 00:33:04.688927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:04.688886    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bba5ed-57d4564ff-dg2lz"]
Oct 31 00:33:05.119271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:05.119240    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f5ea501-6174-4d59-9841-62e7f4084a92" path="/var/lib/kubelet/pods/6f5ea501-6174-4d59-9841-62e7f4084a92/volumes"
Oct 31 00:33:11.308209 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.308177    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"]
Oct 31 00:33:11.308638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.308392    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" podUID="1b48f889-2599-481c-b356-cd3395edb2de" containerName="maas-api" containerID="cri-o://d38e2378c6c5a4b580684df31f4f250673b8b4d96b9997292ecca5bc9df02b31" gracePeriod=30
Oct 31 00:33:11.668323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.668238    4478 generic.go:356] "Generic (PLEG): container finished" podID="1b48f889-2599-481c-b356-cd3395edb2de" containerID="d38e2378c6c5a4b580684df31f4f250673b8b4d96b9997292ecca5bc9df02b31" exitCode=0
Oct 31 00:33:11.668323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.668322    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" event={"ID":"1b48f889-2599-481c-b356-cd3395edb2de","Type":"ContainerDied","Data":"d38e2378c6c5a4b580684df31f4f250673b8b4d96b9997292ecca5bc9df02b31"}
Oct 31 00:33:11.703026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.703000    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:33:11.712516 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.712489    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\") pod \"1b48f889-2599-481c-b356-cd3395edb2de\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") "
Oct 31 00:33:11.712628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.712581    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca\") pod \"1b48f889-2599-481c-b356-cd3395edb2de\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") "
Oct 31 00:33:11.712628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.712607    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs\") pod \"1b48f889-2599-481c-b356-cd3395edb2de\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") "
Oct 31 00:33:11.712755 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.712654    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg\" (UniqueName: \"kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg\") pod \"1b48f889-2599-481c-b356-cd3395edb2de\" (UID: \"1b48f889-2599-481c-b356-cd3395edb2de\") "
Oct 31 00:33:11.713168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.713142    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca" pod "1b48f889-2599-481c-b356-cd3395edb2de" (UID: "1b48f889-2599-481c-b356-cd3395edb2de"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:33:11.714761 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.714731    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls" pod "1b48f889-2599-481c-b356-cd3395edb2de" (UID: "1b48f889-2599-481c-b356-cd3395edb2de"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:33:11.715106 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.715082    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs" pod "1b48f889-2599-481c-b356-cd3395edb2de" (UID: "1b48f889-2599-481c-b356-cd3395edb2de"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:33:11.715268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.715246    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg" pod "1b48f889-2599-481c-b356-cd3395edb2de" (UID: "1b48f889-2599-481c-b356-cd3395edb2de"). InnerVolumeSpecName "kube-api-access-c6rsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:11.821555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.819954    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:11.821555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.819995    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b48f889-2599-481c-b356-cd3395edb2de-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:11.821555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.820012    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1b48f889-2599-481c-b356-cd3395edb2de-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:11.821555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:11.820027    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6rsg\" (UniqueName: \"kubernetes.io/projected/1b48f889-2599-481c-b356-cd3395edb2de-kube-api-access-c6rsg\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:12.676661 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:12.676630    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"
Oct 31 00:33:12.677094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:12.676674    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5" event={"ID":"1b48f889-2599-481c-b356-cd3395edb2de","Type":"ContainerDied","Data":"eb873078312fbbc96a5cf23cb1ee3f3e85a5b3ee07e56792995a6c10b9769453"}
Oct 31 00:33:12.677094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:12.676733    4478 scope.go:122] "RemoveContainer" containerID="d38e2378c6c5a4b580684df31f4f250673b8b4d96b9997292ecca5bc9df02b31"
Oct 31 00:33:12.840801 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:12.840767    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"]
Oct 31 00:33:12.891039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:12.890997    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a4dc2e-bdd7f7cb7-2psc5"]
Oct 31 00:33:13.118154 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:13.118080    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b48f889-2599-481c-b356-cd3395edb2de" path="/var/lib/kubelet/pods/1b48f889-2599-481c-b356-cd3395edb2de/volumes"
Oct 31 00:33:14.765764 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:33:14.765724    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:33:44.757875 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:33:44.757838    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:33:51.022470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:51.022434    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"]
Oct 31 00:33:51.023060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:51.022682    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" containerID="cri-o://ae5a544637257d896d4c0609869b42e96a4e875665c8e4f257c1037b25201bb7" gracePeriod=30
Oct 31 00:33:55.945700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:55.945665    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:33:55.946072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:55.945720    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:33:56.162008 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.161975    4478 generic.go:356] "Generic (PLEG): container finished" podID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerID="ae5a544637257d896d4c0609869b42e96a4e875665c8e4f257c1037b25201bb7" exitCode=0
Oct 31 00:33:56.162154 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.162052    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" event={"ID":"65663a9c-63df-4a76-ad07-26b7a1597b87","Type":"ContainerDied","Data":"ae5a544637257d896d4c0609869b42e96a4e875665c8e4f257c1037b25201bb7"}
Oct 31 00:33:56.394226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.394198    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:33:56.523114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523029    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523072    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523099    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523172    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523216    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523290    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:56.523437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523329    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523373    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:56.523437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523422    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523455    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523526    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo\") pod \"65663a9c-63df-4a76-ad07-26b7a1597b87\" (UID: \"65663a9c-63df-4a76-ad07-26b7a1597b87\") "
Oct 31 00:33:56.523744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523595    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:56.524326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.523922    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:33:56.524326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.524255    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:56.524613 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.524590    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.524693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.524619    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65663a9c-63df-4a76-ad07-26b7a1597b87-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.524693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.524634    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.524693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.524648    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.524693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.524662    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.525795 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.525771    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:33:56.525926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.525826    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "kube-api-access-4hqbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:56.525987 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.525927    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:33:56.526096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.526080    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo" pod "65663a9c-63df-4a76-ad07-26b7a1597b87" (UID: "65663a9c-63df-4a76-ad07-26b7a1597b87"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:33:56.626204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.626173    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.626204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.626200    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqbg\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-kube-api-access-4hqbg\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.626204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.626210    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:56.626471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:56.626219    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65663a9c-63df-4a76-ad07-26b7a1597b87-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:33:57.170120 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:57.170076    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp" event={"ID":"65663a9c-63df-4a76-ad07-26b7a1597b87","Type":"ContainerDied","Data":"ab18b2df76b19c7bec392968717a8163af030496aa4e2127ad96cf6a261bf20b"}
Oct 31 00:33:57.170618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:57.170144    4478 scope.go:122] "RemoveContainer" containerID="ae5a544637257d896d4c0609869b42e96a4e875665c8e4f257c1037b25201bb7"
Oct 31 00:33:57.170618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:57.170091    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"
Oct 31 00:33:57.385189 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:57.385112    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"]
Oct 31 00:33:57.389882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:57.389855    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-9nmtp"]
Oct 31 00:33:59.118466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:33:59.118439    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" path="/var/lib/kubelet/pods/65663a9c-63df-4a76-ad07-26b7a1597b87/volumes"
Oct 31 00:34:00.896612 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.896579    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"]
Oct 31 00:34:00.897191 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897177    4478 cpu_manager.go:414] "containerMap: removing container" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy"
Oct 31 00:34:00.897234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897192    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy"
Oct 31 00:34:00.897234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897221    4478 cpu_manager.go:414] "containerMap: removing container" podUID="1b48f889-2599-481c-b356-cd3395edb2de" containerName="maas-api"
Oct 31 00:34:00.897234 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897226    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b48f889-2599-481c-b356-cd3395edb2de" containerName="maas-api"
Oct 31 00:34:00.897330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897236    4478 cpu_manager.go:414] "containerMap: removing container" podUID="6f5ea501-6174-4d59-9841-62e7f4084a92" containerName="payload-processing"
Oct 31 00:34:00.897330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897241    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f5ea501-6174-4d59-9841-62e7f4084a92" containerName="payload-processing"
Oct 31 00:34:00.897330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897327    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="65663a9c-63df-4a76-ad07-26b7a1597b87" containerName="istio-proxy"
Oct 31 00:34:00.897443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897339    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b48f889-2599-481c-b356-cd3395edb2de" containerName="maas-api"
Oct 31 00:34:00.897443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.897348    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f5ea501-6174-4d59-9841-62e7f4084a92" containerName="payload-processing"
Oct 31 00:34:00.902319 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.902299    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.946075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.946048    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"]
Oct 31 00:34:00.950431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.950375    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:00.984410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984234    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984332    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984359    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984418    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984468    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984564    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984590    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rs8gr\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984689    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:00.984777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:00.984718    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.010418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.010056    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"]
Oct 31 00:34:01.107554 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.107510    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.107723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.107562    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.107723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.107611    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.107723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.107639    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr4gl\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.107723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.107688    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.107953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.107745    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.108789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.108757    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.108987 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.108969    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.109176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.109119    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.109347 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.109327    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.109558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.109537    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.109679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.109663    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.109860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.109845    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.110036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.110019    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rs8gr\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.110162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.110145    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.110334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.110316    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.110536 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.110518    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.110676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.110657    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.113273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.113227    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"]
Oct 31 00:34:01.114067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.114034    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.114250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.114228    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.114625 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.114595    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.114907 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.114887    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.115086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.115057    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.115166 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.115105    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.115442 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.115423    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.171531 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.171439    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.178165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.178131    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rs8gr\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.211704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.211675    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.211854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.211712    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr4gl\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.211854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.211837    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.211991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.211937    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.211991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.211962    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212053    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212306    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212323    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212506    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212514    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212568    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212586    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212733    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.212866 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.212799    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.214236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.214212    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.214584 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.214566    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.219772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.219753    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:01.274944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.274892    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.285589 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.285494    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr4gl\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl\") pod \"e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") " pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.567033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.567001    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:01.655610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.655584    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"]
Oct 31 00:34:01.662037 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.661997    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:34:01.662155 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.662091    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:34:01.662155 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.662134    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:34:01.952624 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.952599    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"]
Oct 31 00:34:01.955084 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:34:01.955052    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfa66e8_6d8a_4814_81c1_6e2d7c685d84.slice/crio-a56dde79e4b90960bed631524aa573edb9233e66c7d192acf1d3a4eade3c662b WatchSource:0}: Error finding container a56dde79e4b90960bed631524aa573edb9233e66c7d192acf1d3a4eade3c662b: Status 404 returned error can't find the container with id a56dde79e4b90960bed631524aa573edb9233e66c7d192acf1d3a4eade3c662b
Oct 31 00:34:01.957747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.957711    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:34:01.957842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.957777    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:34:01.957842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:01.957806    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:34:02.205788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.205687    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" event={"ID":"9b8a74a7-cbc9-46f4-a670-c286729d4f27","Type":"ContainerStarted","Data":"18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679"}
Oct 31 00:34:02.205788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.205762    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" event={"ID":"9b8a74a7-cbc9-46f4-a670-c286729d4f27","Type":"ContainerStarted","Data":"c75c077e5a360097b525a2ade8608c2d73002b9d0bc416c22cf2980ec27d0499"}
Oct 31 00:34:02.207548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.207517    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" event={"ID":"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84","Type":"ContainerStarted","Data":"24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4"}
Oct 31 00:34:02.207702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.207572    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" event={"ID":"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84","Type":"ContainerStarted","Data":"a56dde79e4b90960bed631524aa573edb9233e66c7d192acf1d3a4eade3c662b"}
Oct 31 00:34:02.220865 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.220820    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:02.222697 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.222669    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:02.222813 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.222723    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:02.328087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.328016    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podStartSLOduration=2.327996004 podStartE2EDuration="2.327996004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.324889916 +0000 UTC m=+1875.341554655" watchObservedRunningTime="2026-10-31 00:34:02.327996004 +0000 UTC m=+1875.344660719"
Oct 31 00:34:02.462040 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.461770    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podStartSLOduration=2.461750551 podStartE2EDuration="2.461750551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:34:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:34:02.455053018 +0000 UTC m=+1875.471717731" watchObservedRunningTime="2026-10-31 00:34:02.461750551 +0000 UTC m=+1875.478415265"
Oct 31 00:34:02.569565 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:02.569483    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:03.220935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:03.220904    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:03.221333 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:03.220952    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:03.582674 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:03.582581    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:34:03.582937 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:03.582899    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:34:04.220795 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:04.220764    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:04.220981 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:04.220815    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:04.571460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:04.571136    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:34:04.571460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:04.571195    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:34:05.220335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:05.220273    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:05.220335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:05.220326    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:05.570877 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:05.570768    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:34:05.571543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:05.571509    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:34:06.220093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:06.220064    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:06.220343 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:06.220322    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:06.576465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:06.570740    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:34:06.576465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:06.570812    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:34:07.220085 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:07.220051    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:07.220237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:07.220114    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:07.574826 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:07.574725    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body=
Oct 31 00:34:07.575065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:07.575033    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused"
Oct 31 00:34:08.221116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:08.221069    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Oct 31 00:34:08.221669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:08.221136    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Oct 31 00:34:08.641914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:08.641827    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:08.642715 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:08.642690    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:08.643550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:08.643531    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:34:09.224557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:09.224529    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:09.224937 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:09.224804    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:09.225451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:09.225422    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:34:14.759313 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:34:14.759276    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:34:20.167556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.167520    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"]
Oct 31 00:34:20.167978 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.167757    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72" podUID="d7d6d0ef-9171-403e-b17e-d9b21148241e" containerName="maas-api" containerID="cri-o://f8e82ff022d12196a3da03db7d5ed3dc9f5be5d13b6aa93d9cb42546a536d303" gracePeriod=30
Oct 31 00:34:20.368680 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.368516    4478 generic.go:356] "Generic (PLEG): container finished" podID="d7d6d0ef-9171-403e-b17e-d9b21148241e" containerID="f8e82ff022d12196a3da03db7d5ed3dc9f5be5d13b6aa93d9cb42546a536d303" exitCode=0
Oct 31 00:34:20.368680 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.368555    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72" event={"ID":"d7d6d0ef-9171-403e-b17e-d9b21148241e","Type":"ContainerDied","Data":"f8e82ff022d12196a3da03db7d5ed3dc9f5be5d13b6aa93d9cb42546a536d303"}
Oct 31 00:34:20.637926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.637570    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:34:20.780327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.780241    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c\" (UniqueName: \"kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c\") pod \"d7d6d0ef-9171-403e-b17e-d9b21148241e\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") "
Oct 31 00:34:20.780518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.780332    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca\") pod \"d7d6d0ef-9171-403e-b17e-d9b21148241e\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") "
Oct 31 00:34:20.780592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.780552    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs\") pod \"d7d6d0ef-9171-403e-b17e-d9b21148241e\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") "
Oct 31 00:34:20.780717 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.780699    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\") pod \"d7d6d0ef-9171-403e-b17e-d9b21148241e\" (UID: \"d7d6d0ef-9171-403e-b17e-d9b21148241e\") "
Oct 31 00:34:20.780779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.780736    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca" pod "d7d6d0ef-9171-403e-b17e-d9b21148241e" (UID: "d7d6d0ef-9171-403e-b17e-d9b21148241e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:20.781289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.781264    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d7d6d0ef-9171-403e-b17e-d9b21148241e-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:20.782741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.782712    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c" pod "d7d6d0ef-9171-403e-b17e-d9b21148241e" (UID: "d7d6d0ef-9171-403e-b17e-d9b21148241e"). InnerVolumeSpecName "kube-api-access-b6f2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:20.783185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.783164    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs" pod "d7d6d0ef-9171-403e-b17e-d9b21148241e" (UID: "d7d6d0ef-9171-403e-b17e-d9b21148241e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:20.783591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.783571    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls" pod "d7d6d0ef-9171-403e-b17e-d9b21148241e" (UID: "d7d6d0ef-9171-403e-b17e-d9b21148241e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:34:20.882507 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.882477    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:20.882507 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.882506    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d7d6d0ef-9171-403e-b17e-d9b21148241e-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:20.882696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:20.882519    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6f2c\" (UniqueName: \"kubernetes.io/projected/d7d6d0ef-9171-403e-b17e-d9b21148241e-kube-api-access-b6f2c\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:21.398461 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:21.398081    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72" event={"ID":"d7d6d0ef-9171-403e-b17e-d9b21148241e","Type":"ContainerDied","Data":"d7ffa4fe2b8bb53d170b1ee9cc46c665f7c0de27baf0ce435604c80fd4f797b0"}
Oct 31 00:34:21.398461 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:21.398199    4478 scope.go:122] "RemoveContainer" containerID="f8e82ff022d12196a3da03db7d5ed3dc9f5be5d13b6aa93d9cb42546a536d303"
Oct 31 00:34:21.398461 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:21.398444    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"
Oct 31 00:34:21.563431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:21.563374    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"]
Oct 31 00:34:21.612917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:21.612883    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-835921-668f458884-h4r72"]
Oct 31 00:34:23.120515 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:23.120484    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d6d0ef-9171-403e-b17e-d9b21148241e" path="/var/lib/kubelet/pods/d7d6d0ef-9171-403e-b17e-d9b21148241e/volumes"
Oct 31 00:34:38.460175 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:38.460139    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"]
Oct 31 00:34:41.083927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:41.083887    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"]
Oct 31 00:34:43.461064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.461009    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn" podUID="048088a9-09d2-4410-8f47-f01884d7bfd5" containerName="payload-processing" containerID="cri-o://034d7cc171beb54368182b09ef81b456a972f6c9ddcd6d43184acf9a684458e3" gracePeriod=25
Oct 31 00:34:43.620119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.620082    4478 generic.go:356] "Generic (PLEG): container finished" podID="048088a9-09d2-4410-8f47-f01884d7bfd5" containerID="034d7cc171beb54368182b09ef81b456a972f6c9ddcd6d43184acf9a684458e3" exitCode=0
Oct 31 00:34:43.620265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.620152    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn" event={"ID":"048088a9-09d2-4410-8f47-f01884d7bfd5","Type":"ContainerDied","Data":"034d7cc171beb54368182b09ef81b456a972f6c9ddcd6d43184acf9a684458e3"}
Oct 31 00:34:43.752950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.752925    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:34:43.824821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.824796    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume\") pod \"048088a9-09d2-4410-8f47-f01884d7bfd5\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") "
Oct 31 00:34:43.824991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.824856    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7\" (UniqueName: \"kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7\") pod \"048088a9-09d2-4410-8f47-f01884d7bfd5\" (UID: \"048088a9-09d2-4410-8f47-f01884d7bfd5\") "
Oct 31 00:34:43.825161 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.825138    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume" pod "048088a9-09d2-4410-8f47-f01884d7bfd5" (UID: "048088a9-09d2-4410-8f47-f01884d7bfd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:43.827151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.827127    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7" pod "048088a9-09d2-4410-8f47-f01884d7bfd5" (UID: "048088a9-09d2-4410-8f47-f01884d7bfd5"). InnerVolumeSpecName "kube-api-access-dzbm7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:43.926434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.926373    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048088a9-09d2-4410-8f47-f01884d7bfd5-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:43.926434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:43.926428    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzbm7\" (UniqueName: \"kubernetes.io/projected/048088a9-09d2-4410-8f47-f01884d7bfd5-kube-api-access-dzbm7\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:44.627462 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:44.627429    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"
Oct 31 00:34:44.627462 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:44.627445    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn" event={"ID":"048088a9-09d2-4410-8f47-f01884d7bfd5","Type":"ContainerDied","Data":"2bc8850c771579facd736c50bfa5e4fd4cb39fb7f5b049fcbe997cae6f871d95"}
Oct 31 00:34:44.627974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:44.627544    4478 scope.go:122] "RemoveContainer" containerID="034d7cc171beb54368182b09ef81b456a972f6c9ddcd6d43184acf9a684458e3"
Oct 31 00:34:44.759875 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:34:44.759833    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:34:44.791749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:44.791681    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"]
Oct 31 00:34:44.841011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:44.840982    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-f7c92a85-67769656df-q75qn"]
Oct 31 00:34:45.118413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:45.118312    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048088a9-09d2-4410-8f47-f01884d7bfd5" path="/var/lib/kubelet/pods/048088a9-09d2-4410-8f47-f01884d7bfd5/volumes"
Oct 31 00:34:46.085214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.085177    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz" podUID="942a7b16-b002-48e1-88b3-d08d6c0ea13a" containerName="payload-processing" containerID="cri-o://d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709" gracePeriod=25
Oct 31 00:34:46.371651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.371622    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:34:46.461038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.461009    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume\") pod \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") "
Oct 31 00:34:46.461038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.461046    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7\" (UniqueName: \"kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7\") pod \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\" (UID: \"942a7b16-b002-48e1-88b3-d08d6c0ea13a\") "
Oct 31 00:34:46.461450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.461426    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume" pod "942a7b16-b002-48e1-88b3-d08d6c0ea13a" (UID: "942a7b16-b002-48e1-88b3-d08d6c0ea13a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:46.461599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.461584    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/942a7b16-b002-48e1-88b3-d08d6c0ea13a-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:46.463353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.463325    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7" pod "942a7b16-b002-48e1-88b3-d08d6c0ea13a" (UID: "942a7b16-b002-48e1-88b3-d08d6c0ea13a"). InnerVolumeSpecName "kube-api-access-kx6r7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:46.562965 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.562926    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx6r7\" (UniqueName: \"kubernetes.io/projected/942a7b16-b002-48e1-88b3-d08d6c0ea13a-kube-api-access-kx6r7\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:46.641730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.641645    4478 generic.go:356] "Generic (PLEG): container finished" podID="942a7b16-b002-48e1-88b3-d08d6c0ea13a" containerID="d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709" exitCode=0
Oct 31 00:34:46.641730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.641694    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"
Oct 31 00:34:46.641967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.641727    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz" event={"ID":"942a7b16-b002-48e1-88b3-d08d6c0ea13a","Type":"ContainerDied","Data":"d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709"}
Oct 31 00:34:46.641967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.641784    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz" event={"ID":"942a7b16-b002-48e1-88b3-d08d6c0ea13a","Type":"ContainerDied","Data":"23eb08f3c9e4d1b7b9f1e38dfb4b856351c5c90bf65a2ee115a710acb4f004d5"}
Oct 31 00:34:46.641967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.641815    4478 scope.go:122] "RemoveContainer" containerID="d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709"
Oct 31 00:34:46.652471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.652451    4478 scope.go:122] "RemoveContainer" containerID="d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709"
Oct 31 00:34:46.652757 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:34:46.652737    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709\": container with ID starting with d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709 not found: ID does not exist" containerID="d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709"
Oct 31 00:34:46.652870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.652765    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709"} err="failed to get container status \"d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709\": rpc error: code = NotFound desc = could not find container \"d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709\": container with ID starting with d03542f0fba74d8e699293ba75a41fb448499bbecbb30900a5159c8c03530709 not found: ID does not exist"
Oct 31 00:34:46.806771 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.806736    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"]
Oct 31 00:34:46.810494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:46.810465    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2a59a7-7fb7b7bb7-rcnkz"]
Oct 31 00:34:47.118568 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:47.118539    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="942a7b16-b002-48e1-88b3-d08d6c0ea13a" path="/var/lib/kubelet/pods/942a7b16-b002-48e1-88b3-d08d6c0ea13a/volumes"
Oct 31 00:34:50.345308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:50.345274    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"]
Oct 31 00:34:50.345724 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:50.345521    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" containerID="cri-o://e31859904895a8a7f295742ae59092824f3cfeb71800e4e9ef38c82219be376a" gracePeriod=30
Oct 31 00:34:50.395294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:50.395257    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"]
Oct 31 00:34:50.395547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:50.395520    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" containerID="cri-o://13eb0e55154f6a3598ee594d4c8c6fe4594f9c45772c315e0168e5d5ebc76737" gracePeriod=30
Oct 31 00:34:50.398369 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:50.398344    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"]
Oct 31 00:34:50.398595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:50.398567    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" containerID="cri-o://8072c9a02684e826ecf2fb4815753d628c898ed0cbe334e7c59bb74d6bfc0aa4" gracePeriod=30
Oct 31 00:34:54.108016 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:54.101804    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-4znwv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:34:54.108016 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:54.101872    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:34:54.119686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:54.119633    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-cpsx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:34:54.119991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:54.119964    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:34:54.191081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:54.191048    4478 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58c6585447-j6ldv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:34:54.191231 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:54.191120    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:34:55.711555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.711518    4478 generic.go:356] "Generic (PLEG): container finished" podID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerID="8072c9a02684e826ecf2fb4815753d628c898ed0cbe334e7c59bb74d6bfc0aa4" exitCode=0
Oct 31 00:34:55.711982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.711662    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" event={"ID":"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8","Type":"ContainerDied","Data":"8072c9a02684e826ecf2fb4815753d628c898ed0cbe334e7c59bb74d6bfc0aa4"}
Oct 31 00:34:55.713133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.713106    4478 generic.go:356] "Generic (PLEG): container finished" podID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerID="e31859904895a8a7f295742ae59092824f3cfeb71800e4e9ef38c82219be376a" exitCode=0
Oct 31 00:34:55.713252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.713180    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" event={"ID":"3857a068-f40e-47be-a7a8-7c9e71c46daf","Type":"ContainerDied","Data":"e31859904895a8a7f295742ae59092824f3cfeb71800e4e9ef38c82219be376a"}
Oct 31 00:34:55.713252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.713215    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv" event={"ID":"3857a068-f40e-47be-a7a8-7c9e71c46daf","Type":"ContainerDied","Data":"06fc5a016bbff112a1436e3bceeac78c664fe76abb2f3888968ba2140f3cab69"}
Oct 31 00:34:55.713252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.713231    4478 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06fc5a016bbff112a1436e3bceeac78c664fe76abb2f3888968ba2140f3cab69"
Oct 31 00:34:55.714618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.714598    4478 generic.go:356] "Generic (PLEG): container finished" podID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerID="13eb0e55154f6a3598ee594d4c8c6fe4594f9c45772c315e0168e5d5ebc76737" exitCode=0
Oct 31 00:34:55.714720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.714647    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" event={"ID":"19ddd893-2592-4ec8-b8e8-8b4030d831a8","Type":"ContainerDied","Data":"13eb0e55154f6a3598ee594d4c8c6fe4594f9c45772c315e0168e5d5ebc76737"}
Oct 31 00:34:55.727291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.727268    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:34:55.783870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.783844    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:34:55.792755 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.792733    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:34:55.803702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.803681    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.803815 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.803803    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.803859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.803847    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.803894 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.803871    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.803928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.803896    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.803928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.803912    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.804258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804029    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.804258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804099    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.804258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804130    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket\") pod \"3857a068-f40e-47be-a7a8-7c9e71c46daf\" (UID: \"3857a068-f40e-47be-a7a8-7c9e71c46daf\") "
Oct 31 00:34:55.804258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804113    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.804258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804159    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:55.804258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804198    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804320    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804448    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804728    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804742    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804751    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3857a068-f40e-47be-a7a8-7c9e71c46daf-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804761    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.804859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.804775    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.806111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.806092    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:34:55.806373 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.806348    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "kube-api-access-nssdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:55.806645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.806629    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.806690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.806648    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token" pod "3857a068-f40e-47be-a7a8-7c9e71c46daf" (UID: "3857a068-f40e-47be-a7a8-7c9e71c46daf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:55.905364 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905302    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905364 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905342    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905396    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905429    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905455    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905488    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905514    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905538    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905559    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905581    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905604    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905624    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905648    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905670    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905693    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.905779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905765    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.906189 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905782    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp\") pod \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\" (UID: \"19ddd893-2592-4ec8-b8e8-8b4030d831a8\") "
Oct 31 00:34:55.906189 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.905813    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert\") pod \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\" (UID: \"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8\") "
Oct 31 00:34:55.906285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906224    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.906285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906235    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.906285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906246    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.906285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906255    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nssdh\" (UniqueName: \"kubernetes.io/projected/3857a068-f40e-47be-a7a8-7c9e71c46daf-kube-api-access-nssdh\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906575    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906600    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906650    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906721    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.906735    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.907033    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.907164    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:34:55.907497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.907455    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.908171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.907662    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.909585 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.909561    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.909806 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.909747    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.909914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.909849    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "kube-api-access-bhgvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:55.909914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.909878    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:55.910045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.909940    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "kube-api-access-w22mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:55.910774 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.910747    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:34:55.911095 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.911072    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:34:55.911956 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.911931    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo" pod "19ddd893-2592-4ec8-b8e8-8b4030d831a8" (UID: "19ddd893-2592-4ec8-b8e8-8b4030d831a8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:34:55.912053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:55.911980    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token" pod "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" (UID: "be3f7644-c2cd-43fa-81f6-ec257f7eb5b8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:34:56.020092 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020056    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020092 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020093    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020092 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020107    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020123    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020137    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhgvd\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-kube-api-access-bhgvd\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020151    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020163    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020175    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020189    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020203    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020218    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020230    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w22mp\" (UniqueName: \"kubernetes.io/projected/19ddd893-2592-4ec8-b8e8-8b4030d831a8-kube-api-access-w22mp\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020244    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020257    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020269    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020284    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020297    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.020476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.020308    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19ddd893-2592-4ec8-b8e8-8b4030d831a8-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:34:56.722663 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.722628    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"
Oct 31 00:34:56.723118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.722658    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2" event={"ID":"19ddd893-2592-4ec8-b8e8-8b4030d831a8","Type":"ContainerDied","Data":"802d21401c80066aa6fd979eb07cdbfd6d818b5f8cc2a3a315780bab6f85ea02"}
Oct 31 00:34:56.723118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.722720    4478 scope.go:122] "RemoveContainer" containerID="13eb0e55154f6a3598ee594d4c8c6fe4594f9c45772c315e0168e5d5ebc76737"
Oct 31 00:34:56.724304 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.724282    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"
Oct 31 00:34:56.724439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.724299    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv" event={"ID":"be3f7644-c2cd-43fa-81f6-ec257f7eb5b8","Type":"ContainerDied","Data":"cec76eaadfe11a0700823a6ad14bac82545155d31188ea3330d0bbc3a353047f"}
Oct 31 00:34:56.724504 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.724486    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"
Oct 31 00:34:56.735730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.735710    4478 scope.go:122] "RemoveContainer" containerID="8072c9a02684e826ecf2fb4815753d628c898ed0cbe334e7c59bb74d6bfc0aa4"
Oct 31 00:34:56.888111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.888077    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"]
Oct 31 00:34:56.891527 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:56.891504    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-j6ldv"]
Oct 31 00:34:57.001220 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:57.001127    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"]
Oct 31 00:34:57.004434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:57.004411    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-4znwv"]
Oct 31 00:34:57.136100 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:57.136063    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" path="/var/lib/kubelet/pods/3857a068-f40e-47be-a7a8-7c9e71c46daf/volumes"
Oct 31 00:34:57.136536 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:57.136520    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" path="/var/lib/kubelet/pods/be3f7644-c2cd-43fa-81f6-ec257f7eb5b8/volumes"
Oct 31 00:34:57.136799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:57.136787    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"]
Oct 31 00:34:57.136842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:57.136814    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58c6585447-cpsx2"]
Oct 31 00:34:59.118025 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:34:59.117955    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" path="/var/lib/kubelet/pods/19ddd893-2592-4ec8-b8e8-8b4030d831a8/volumes"
Oct 31 00:35:14.912990 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:35:14.912947    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:35:25.754043 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:25.754003    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"]
Oct 31 00:35:25.754586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:25.754239    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" podUID="3e03964e-6172-468c-a123-fea4cb43564a" containerName="maas-api" containerID="cri-o://bbb10fe67e874c95b0a073a14e8e87e6051e1f4babb184c7f04de4232fef47d9" gracePeriod=30
Oct 31 00:35:26.072472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.072287    4478 generic.go:356] "Generic (PLEG): container finished" podID="3e03964e-6172-468c-a123-fea4cb43564a" containerID="bbb10fe67e874c95b0a073a14e8e87e6051e1f4babb184c7f04de4232fef47d9" exitCode=0
Oct 31 00:35:26.072472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.072359    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" event={"ID":"3e03964e-6172-468c-a123-fea4cb43564a","Type":"ContainerDied","Data":"bbb10fe67e874c95b0a073a14e8e87e6051e1f4babb184c7f04de4232fef47d9"}
Oct 31 00:35:26.126009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.125980    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:35:26.177407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.177023    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca\") pod \"3e03964e-6172-468c-a123-fea4cb43564a\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") "
Oct 31 00:35:26.177407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.177082    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st\" (UniqueName: \"kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st\") pod \"3e03964e-6172-468c-a123-fea4cb43564a\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") "
Oct 31 00:35:26.177407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.177263    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs\") pod \"3e03964e-6172-468c-a123-fea4cb43564a\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") "
Oct 31 00:35:26.177407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.177293    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\") pod \"3e03964e-6172-468c-a123-fea4cb43564a\" (UID: \"3e03964e-6172-468c-a123-fea4cb43564a\") "
Oct 31 00:35:26.205432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.200573    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca" pod "3e03964e-6172-468c-a123-fea4cb43564a" (UID: "3e03964e-6172-468c-a123-fea4cb43564a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:26.235510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.231663    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls" pod "3e03964e-6172-468c-a123-fea4cb43564a" (UID: "3e03964e-6172-468c-a123-fea4cb43564a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:26.236131 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.236091    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st" pod "3e03964e-6172-468c-a123-fea4cb43564a" (UID: "3e03964e-6172-468c-a123-fea4cb43564a"). InnerVolumeSpecName "kube-api-access-2r4st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:26.284665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.284626    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:26.284665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.284668    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e03964e-6172-468c-a123-fea4cb43564a-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:26.284877 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.284686    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r4st\" (UniqueName: \"kubernetes.io/projected/3e03964e-6172-468c-a123-fea4cb43564a-kube-api-access-2r4st\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:26.293250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.293209    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs" pod "3e03964e-6172-468c-a123-fea4cb43564a" (UID: "3e03964e-6172-468c-a123-fea4cb43564a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:35:26.385586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:26.385507    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3e03964e-6172-468c-a123-fea4cb43564a-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:27.080190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:27.080111    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"
Oct 31 00:35:27.080190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:27.080142    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj" event={"ID":"3e03964e-6172-468c-a123-fea4cb43564a","Type":"ContainerDied","Data":"cb9cc3d2c5a4da439c73c49161612fb5d92c8a1a6e2e795623db81dc15ec9e3e"}
Oct 31 00:35:27.080696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:27.080203    4478 scope.go:122] "RemoveContainer" containerID="bbb10fe67e874c95b0a073a14e8e87e6051e1f4babb184c7f04de4232fef47d9"
Oct 31 00:35:27.259739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:27.259707    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"]
Oct 31 00:35:27.262286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:27.262265    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2ccdde-55c9f74458-qzqgj"]
Oct 31 00:35:29.117861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:29.117832    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e03964e-6172-468c-a123-fea4cb43564a" path="/var/lib/kubelet/pods/3e03964e-6172-468c-a123-fea4cb43564a/volumes"
Oct 31 00:35:44.758973 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:35:44.758937    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:35:47.460190 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:47.460162    4478 scope.go:122] "RemoveContainer" containerID="e31859904895a8a7f295742ae59092824f3cfeb71800e4e9ef38c82219be376a"
Oct 31 00:35:49.943948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:49.943911    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"]
Oct 31 00:35:49.944411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:49.944139    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy" containerID="cri-o://a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55" gracePeriod=30
Oct 31 00:35:49.960125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:49.960094    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"]
Oct 31 00:35:49.960339 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:49.960296    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy" containerID="cri-o://695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0" gracePeriod=30
Oct 31 00:35:55.235024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.234961    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:35:55.269335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.269312    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:35:55.321005 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.320981    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.321125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321017    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.321125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321061    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.321125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321081    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.321125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321102    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.321333 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321152    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.321422 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321351    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.321494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321433    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.321538 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321490    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.321538 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321496    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.321646 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321627    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:55.321915 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321889    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:35:55.322013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.321993    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.322059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322037    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.322115 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322072    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.322171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322127    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.322229 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322185    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.322287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322229    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.322731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322352    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.322731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322466    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.322731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322498    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy\") pod \"2697501d-54c8-4889-9d8c-3af0f7258922\" (UID: \"2697501d-54c8-4889-9d8c-3af0f7258922\") "
Oct 31 00:35:55.322731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322533    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.322731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322673    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.323124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322851    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.323124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322879    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.323124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.322975    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs\") pod \"1d11bad4-844a-428f-b295-b7500cdeb3d0\" (UID: \"1d11bad4-844a-428f-b295-b7500cdeb3d0\") "
Oct 31 00:35:55.323124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323076    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.323439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323328    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.323729 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323706    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:35:55.323803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323620    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.323803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323756    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.323803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323771    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.323803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323784    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2697501d-54c8-4889-9d8c-3af0f7258922-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.323803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323798    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.324076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323811    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.324076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323825    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.324076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323838    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d11bad4-844a-428f-b295-b7500cdeb3d0-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.324076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323852    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.324076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323865    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.324076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.323902    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.324649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.324624    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "kube-api-access-6z4pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:55.324861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.324839    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:55.325062 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.325031    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:35:55.325521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.325502    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "kube-api-access-sf9jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:55.325609 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.325571    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token" pod "2697501d-54c8-4889-9d8c-3af0f7258922" (UID: "2697501d-54c8-4889-9d8c-3af0f7258922"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:35:55.325841 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.325819    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo" pod "1d11bad4-844a-428f-b295-b7500cdeb3d0" (UID: "1d11bad4-844a-428f-b295-b7500cdeb3d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:35:55.348530 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.348505    4478 generic.go:356] "Generic (PLEG): container finished" podID="2697501d-54c8-4889-9d8c-3af0f7258922" containerID="695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0" exitCode=0
Oct 31 00:35:55.348638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.348550    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"
Oct 31 00:35:55.348638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.348582    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" event={"ID":"2697501d-54c8-4889-9d8c-3af0f7258922","Type":"ContainerDied","Data":"695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0"}
Oct 31 00:35:55.348638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.348635    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck" event={"ID":"2697501d-54c8-4889-9d8c-3af0f7258922","Type":"ContainerDied","Data":"6e00d137e7c13a13af70ff4a9aecf3c086be51bcaee165d90894f9996cf9d06c"}
Oct 31 00:35:55.348768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.348661    4478 scope.go:122] "RemoveContainer" containerID="695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0"
Oct 31 00:35:55.349889 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.349872    4478 generic.go:356] "Generic (PLEG): container finished" podID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerID="a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55" exitCode=0
Oct 31 00:35:55.349970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.349926    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"
Oct 31 00:35:55.349970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.349943    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" event={"ID":"1d11bad4-844a-428f-b295-b7500cdeb3d0","Type":"ContainerDied","Data":"a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55"}
Oct 31 00:35:55.350049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.349988    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj" event={"ID":"1d11bad4-844a-428f-b295-b7500cdeb3d0","Type":"ContainerDied","Data":"ac3e3ae9a1bee138edc2aa56bef1a6d733627dcbbb30d7a0c9e2a471e0e30e9f"}
Oct 31 00:35:55.360174 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.360153    4478 scope.go:122] "RemoveContainer" containerID="695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0"
Oct 31 00:35:55.360499 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:35:55.360472    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0\": container with ID starting with 695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0 not found: ID does not exist" containerID="695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0"
Oct 31 00:35:55.360574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.360511    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0"} err="failed to get container status \"695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0\": rpc error: code = NotFound desc = could not find container \"695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0\": container with ID starting with 695a9378c9b6137c10176dbfd75d80a1369db2a9d9ccb75d8f6fe49ff21a74a0 not found: ID does not exist"
Oct 31 00:35:55.360574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.360535    4478 scope.go:122] "RemoveContainer" containerID="a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55"
Oct 31 00:35:55.370516 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.370495    4478 scope.go:122] "RemoveContainer" containerID="a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55"
Oct 31 00:35:55.370778 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:35:55.370763    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55\": container with ID starting with a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55 not found: ID does not exist" containerID="a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55"
Oct 31 00:35:55.370828 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.370784    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55"} err="failed to get container status \"a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55\": rpc error: code = NotFound desc = could not find container \"a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55\": container with ID starting with a76f15f0aee49b83985f51889abd794f5365ca0131a9604615f2de9d63d77d55 not found: ID does not exist"
Oct 31 00:35:55.425080 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425054    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425080 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425079    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6z4pj\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-kube-api-access-6z4pj\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425090    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425098    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9jl\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-kube-api-access-sf9jl\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425108    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2697501d-54c8-4889-9d8c-3af0f7258922-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425117    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2697501d-54c8-4889-9d8c-3af0f7258922-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425127    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d11bad4-844a-428f-b295-b7500cdeb3d0-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.425292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.425137    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2697501d-54c8-4889-9d8c-3af0f7258922-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:35:55.516113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.516042    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"]
Oct 31 00:35:55.525153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.525126    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-m99jj"]
Oct 31 00:35:55.701760 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.701724    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"]
Oct 31 00:35:55.706926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:55.706900    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-68pck"]
Oct 31 00:35:57.119504 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:57.119473    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" path="/var/lib/kubelet/pods/1d11bad4-844a-428f-b295-b7500cdeb3d0/volumes"
Oct 31 00:35:57.120124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:35:57.120105    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" path="/var/lib/kubelet/pods/2697501d-54c8-4889-9d8c-3af0f7258922/volumes"
Oct 31 00:36:05.978945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:05.978914    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"]
Oct 31 00:36:05.979471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:05.979130    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" containerID="cri-o://9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e" gracePeriod=30
Oct 31 00:36:06.000140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:06.000104    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"]
Oct 31 00:36:06.000429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:06.000373    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" containerID="cri-o://24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4" gracePeriod=30
Oct 31 00:36:06.005608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:06.005580    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"]
Oct 31 00:36:06.005853 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:06.005822    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" containerID="cri-o://18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679" gracePeriod=30
Oct 31 00:36:06.737320 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:06.737281    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:36:06.737579 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:06.737361    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:36:07.028297 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.028212    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"]
Oct 31 00:36:07.028707 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.028445    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy" containerID="cri-o://ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e" gracePeriod=30
Oct 31 00:36:07.976142 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976100    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"]
Oct 31 00:36:07.976708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976691    4478 cpu_manager.go:414] "containerMap: removing container" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy"
Oct 31 00:36:07.976708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976707    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976719    4478 cpu_manager.go:414] "containerMap: removing container" podUID="048088a9-09d2-4410-8f47-f01884d7bfd5" containerName="payload-processing"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976724    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048088a9-09d2-4410-8f47-f01884d7bfd5" containerName="payload-processing"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976737    4478 cpu_manager.go:414] "containerMap: removing container" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976742    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976751    4478 cpu_manager.go:414] "containerMap: removing container" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976756    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976775    4478 cpu_manager.go:414] "containerMap: removing container" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976780    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976786    4478 cpu_manager.go:414] "containerMap: removing container" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976791    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976798    4478 cpu_manager.go:414] "containerMap: removing container" podUID="3e03964e-6172-468c-a123-fea4cb43564a" containerName="maas-api"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976802    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e03964e-6172-468c-a123-fea4cb43564a" containerName="maas-api"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976808    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d7d6d0ef-9171-403e-b17e-d9b21148241e" containerName="maas-api"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976813    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d6d0ef-9171-403e-b17e-d9b21148241e" containerName="maas-api"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976824    4478 cpu_manager.go:414] "containerMap: removing container" podUID="942a7b16-b002-48e1-88b3-d08d6c0ea13a" containerName="payload-processing"
Oct 31 00:36:07.976858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976828    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="942a7b16-b002-48e1-88b3-d08d6c0ea13a" containerName="payload-processing"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976901    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d6d0ef-9171-403e-b17e-d9b21148241e" containerName="maas-api"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976909    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="942a7b16-b002-48e1-88b3-d08d6c0ea13a" containerName="payload-processing"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976917    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="3857a068-f40e-47be-a7a8-7c9e71c46daf" containerName="istio-proxy"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976922    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="048088a9-09d2-4410-8f47-f01884d7bfd5" containerName="payload-processing"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976930    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3f7644-c2cd-43fa-81f6-ec257f7eb5b8" containerName="istio-proxy"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976944    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d11bad4-844a-428f-b295-b7500cdeb3d0" containerName="istio-proxy"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976951    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ddd893-2592-4ec8-b8e8-8b4030d831a8" containerName="istio-proxy"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976956    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="2697501d-54c8-4889-9d8c-3af0f7258922" containerName="istio-proxy"
Oct 31 00:36:07.977644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.976962    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e03964e-6172-468c-a123-fea4cb43564a" containerName="maas-api"
Oct 31 00:36:07.981879 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:07.981861    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.039945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.039918    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-752be8-openshift-default-dockercfg-2jkp2\""
Oct 31 00:36:08.043675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.043642    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.043888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.043692    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b542c\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.043888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.043733    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.043888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.043822    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.044108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.043921    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.044108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.043951    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.044212 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.044158    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.044266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.044215    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.044266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.044241    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.089899 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.089868    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"]
Oct 31 00:36:08.145267 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145233    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145315    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145338    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145394    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145436    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145460    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145526    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145550    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b542c\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145585    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.145892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.145790    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.146054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.146034    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.146180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.146152    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.146275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.146251    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.146275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.146267    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.148062 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.148039    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.148771 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.148752    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.226601 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.225560    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b542c\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.241414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.239164    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token\") pod \"e2e-api-a-752be8-openshift-default-79688d7695-mvqqp\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") " pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.296647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.296599    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:08.645595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.645512    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:36:08.645595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.645567    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:36:08.733440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.733409    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"]
Oct 31 00:36:08.735877 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:36:08.735849    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079636e4_f2e7_4c52_8bc7_880d0149f24c.slice/crio-c08fd6e54793d06569fdfa30f1369a24c9f9f75032a539747fbdc2903ae99c66 WatchSource:0}: Error finding container c08fd6e54793d06569fdfa30f1369a24c9f9f75032a539747fbdc2903ae99c66: Status 404 returned error can't find the container with id c08fd6e54793d06569fdfa30f1369a24c9f9f75032a539747fbdc2903ae99c66
Oct 31 00:36:08.738856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.738814    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:36:08.738963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.738889    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:36:08.738963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:08.738923    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:36:09.227216 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:09.227181    4478 patch_prober.go:29] interesting pod/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:36:09.227723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:09.227243    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:36:09.516116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:09.516025    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" event={"ID":"079636e4-f2e7-4c52-8bc7-880d0149f24c","Type":"ContainerStarted","Data":"edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"}
Oct 31 00:36:09.516116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:09.516076    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" event={"ID":"079636e4-f2e7-4c52-8bc7-880d0149f24c","Type":"ContainerStarted","Data":"c08fd6e54793d06569fdfa30f1369a24c9f9f75032a539747fbdc2903ae99c66"}
Oct 31 00:36:09.638511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:09.638442    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podStartSLOduration=2.638423296 podStartE2EDuration="2.638423296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:09.635770884 +0000 UTC m=+2002.652435619" watchObservedRunningTime="2026-10-31 00:36:09.638423296 +0000 UTC m=+2002.655088008"
Oct 31 00:36:10.297182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:10.297143    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:10.303835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:10.303809    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:10.528249 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:10.528213    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:10.529188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:10.529164    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:36:11.328111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.328085    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:36:11.351789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.351764    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:36:11.355179 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.355163    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:36:11.425408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425354    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.425545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425430    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.425545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425465    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.425545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425484    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.425545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425509    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.425545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425535    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425573    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425603    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425614    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425630    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425689    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425721    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.425776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425759    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.425983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425783    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.425983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425816    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.425983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425843    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.425983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425893    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.425983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425948    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.425983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425842    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425871    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.425968    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426008    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426059    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426109    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426167    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426181    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426114    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.426269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426237    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426282    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426314    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426341    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426396    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426426    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426456    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl\") pod \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\" (UID: \"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426562    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token\") pod \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\" (UID: \"9b8a74a7-cbc9-46f4-a670-c286729d4f27\") "
Oct 31 00:36:11.426940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426724    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy\") pod \"4a696228-3344-41d5-8f66-6e0f12e86723\" (UID: \"4a696228-3344-41d5-8f66-6e0f12e86723\") "
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427364    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427423    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427438    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427454    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a696228-3344-41d5-8f66-6e0f12e86723-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427469    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427482    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427498    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427549    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427567    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426425    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426436    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.426616    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427360    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.428094 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.427565    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.428938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.428159    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.429269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.429231    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.429374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.429278    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "kube-api-access-ngnks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:11.429501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.429467    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:36:11.430063 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.430039    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:11.430182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.430077    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "kube-api-access-rs8gr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:11.430429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.430365    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.430658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.430591    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:36:11.430802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.430784    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy" pod "4a696228-3344-41d5-8f66-6e0f12e86723" (UID: "4a696228-3344-41d5-8f66-6e0f12e86723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:11.431002 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.430986    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:11.431281 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.431256    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:11.431455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.431435    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo" pod "9b8a74a7-cbc9-46f4-a670-c286729d4f27" (UID: "9b8a74a7-cbc9-46f4-a670-c286729d4f27"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:36:11.431592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.431579    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl" pod "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" (UID: "cbfa66e8-6d8a-4814-81c1-6e2d7c685d84"). InnerVolumeSpecName "kube-api-access-gr4gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:11.529006 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.528976    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rs8gr\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-kube-api-access-rs8gr\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529006 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529004    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529018    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529028    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529038    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529046    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529055    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529062    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a696228-3344-41d5-8f66-6e0f12e86723-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529071    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529081    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529089    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr4gl\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-kube-api-access-gr4gl\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529097    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b8a74a7-cbc9-46f4-a670-c286729d4f27-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529104    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529112    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529120    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529128    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngnks\" (UniqueName: \"kubernetes.io/projected/4a696228-3344-41d5-8f66-6e0f12e86723-kube-api-access-ngnks\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529136    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.529263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.529144    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a696228-3344-41d5-8f66-6e0f12e86723-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:11.537123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.537090    4478 generic.go:356] "Generic (PLEG): container finished" podID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerID="18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679" exitCode=0
Oct 31 00:36:11.537261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.537142    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"
Oct 31 00:36:11.537261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.537182    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" event={"ID":"9b8a74a7-cbc9-46f4-a670-c286729d4f27","Type":"ContainerDied","Data":"18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679"}
Oct 31 00:36:11.537410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.537329    4478 scope.go:122] "RemoveContainer" containerID="18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679"
Oct 31 00:36:11.537781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.537246    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm" event={"ID":"9b8a74a7-cbc9-46f4-a670-c286729d4f27","Type":"ContainerDied","Data":"c75c077e5a360097b525a2ade8608c2d73002b9d0bc416c22cf2980ec27d0499"}
Oct 31 00:36:11.540548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.540224    4478 generic.go:356] "Generic (PLEG): container finished" podID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerID="24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4" exitCode=0
Oct 31 00:36:11.540548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.540338    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" event={"ID":"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84","Type":"ContainerDied","Data":"24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4"}
Oct 31 00:36:11.540548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.540441    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn" event={"ID":"cbfa66e8-6d8a-4814-81c1-6e2d7c685d84","Type":"ContainerDied","Data":"a56dde79e4b90960bed631524aa573edb9233e66c7d192acf1d3a4eade3c662b"}
Oct 31 00:36:11.541203 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.540551    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"
Oct 31 00:36:11.542787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.542760    4478 generic.go:356] "Generic (PLEG): container finished" podID="4a696228-3344-41d5-8f66-6e0f12e86723" containerID="9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e" exitCode=0
Oct 31 00:36:11.543838 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.543805    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"
Oct 31 00:36:11.543968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.543800    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" event={"ID":"4a696228-3344-41d5-8f66-6e0f12e86723","Type":"ContainerDied","Data":"9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e"}
Oct 31 00:36:11.544083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.544061    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l" event={"ID":"4a696228-3344-41d5-8f66-6e0f12e86723","Type":"ContainerDied","Data":"13ec9cc36c919931efee5445f4b6b03e327460ce37d39eef0e078a29b9a50b17"}
Oct 31 00:36:11.554279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.554248    4478 scope.go:122] "RemoveContainer" containerID="18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679"
Oct 31 00:36:11.554586 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:11.554567    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679\": container with ID starting with 18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679 not found: ID does not exist" containerID="18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679"
Oct 31 00:36:11.554653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.554595    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679"} err="failed to get container status \"18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679\": rpc error: code = NotFound desc = could not find container \"18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679\": container with ID starting with 18a93526d849dab6791c291add5a74351b477d2346140c8c025250f8c65bc679 not found: ID does not exist"
Oct 31 00:36:11.554653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.554616    4478 scope.go:122] "RemoveContainer" containerID="24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4"
Oct 31 00:36:11.563998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.563983    4478 scope.go:122] "RemoveContainer" containerID="24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4"
Oct 31 00:36:11.564255 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:11.564238    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4\": container with ID starting with 24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4 not found: ID does not exist" containerID="24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4"
Oct 31 00:36:11.564302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.564262    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4"} err="failed to get container status \"24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4\": rpc error: code = NotFound desc = could not find container \"24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4\": container with ID starting with 24077542ba82acfb6b69002207ab2a240c0cbe5cc051b8876160cb816805cdf4 not found: ID does not exist"
Oct 31 00:36:11.564302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.564278    4478 scope.go:122] "RemoveContainer" containerID="9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e"
Oct 31 00:36:11.574898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.574873    4478 scope.go:122] "RemoveContainer" containerID="9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e"
Oct 31 00:36:11.575217 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:11.575179    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e\": container with ID starting with 9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e not found: ID does not exist" containerID="9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e"
Oct 31 00:36:11.575310 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.575226    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e"} err="failed to get container status \"9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e\": rpc error: code = NotFound desc = could not find container \"9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e\": container with ID starting with 9f5a1582598a08f9c079adbaf36b138c03cbbacc0b980c5331236fd6b202260e not found: ID does not exist"
Oct 31 00:36:11.832165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.806731    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"]
Oct 31 00:36:11.832165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.811790    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-2rckm"]
Oct 31 00:36:11.937321 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.937292    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"]
Oct 31 00:36:11.941414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:11.941356    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-h24zn"]
Oct 31 00:36:12.051065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.051035    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"]
Oct 31 00:36:12.055833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.055811    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-f7c92a85-openshift-default-59bf496589-db92l"]
Oct 31 00:36:12.157252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.157221    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"]
Oct 31 00:36:12.157525 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.157497    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" containerID="cri-o://05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88" gracePeriod=30
Oct 31 00:36:12.317969 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.317943    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:36:12.341490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341461    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341507    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341571    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341589    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341648    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341677    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341701    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341801    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341840    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs\") pod \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\" (UID: \"b169c664-7798-4ef4-bdfa-efbadc4ccfd4\") "
Oct 31 00:36:12.341884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341855    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:12.342323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341871    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:12.342323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.341979    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:12.342323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342114    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:12.342323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342256    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:12.342730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342689    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.342730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342711    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.342730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342727    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.342905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342744    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.342905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.342758    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.344093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.344059    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "kube-api-access-f848l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:12.344194 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.344097    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:12.344194 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.344131    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:36:12.344194 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.344131    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token" pod "b169c664-7798-4ef4-bdfa-efbadc4ccfd4" (UID: "b169c664-7798-4ef4-bdfa-efbadc4ccfd4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:12.444397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.444352    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.444397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.444400    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.444642 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.444414    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.444642 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.444427    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f848l\" (UniqueName: \"kubernetes.io/projected/b169c664-7798-4ef4-bdfa-efbadc4ccfd4-kube-api-access-f848l\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:12.578655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.578558    4478 generic.go:356] "Generic (PLEG): container finished" podID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerID="ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e" exitCode=0
Oct 31 00:36:12.579570 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.578698    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" event={"ID":"b169c664-7798-4ef4-bdfa-efbadc4ccfd4","Type":"ContainerDied","Data":"ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e"}
Oct 31 00:36:12.579570 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.578754    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7" event={"ID":"b169c664-7798-4ef4-bdfa-efbadc4ccfd4","Type":"ContainerDied","Data":"2eeedddea42b199fa2b54dba7068724bcd8d879d82bab9fb231ed97c05b3984a"}
Oct 31 00:36:12.579570 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.578794    4478 scope.go:122] "RemoveContainer" containerID="ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e"
Oct 31 00:36:12.579570 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.578811    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"
Oct 31 00:36:12.593077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.593049    4478 scope.go:122] "RemoveContainer" containerID="ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e"
Oct 31 00:36:12.593536 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:12.593513    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e\": container with ID starting with ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e not found: ID does not exist" containerID="ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e"
Oct 31 00:36:12.593641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.593550    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e"} err="failed to get container status \"ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e\": rpc error: code = NotFound desc = could not find container \"ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e\": container with ID starting with ac500e1d47b40751ece3b1d6285e103bcc8a6313ea60b9145dbf0733d0ba442e not found: ID does not exist"
Oct 31 00:36:12.791632 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.791555    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"]
Oct 31 00:36:12.836457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:12.836414    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lrcx7"]
Oct 31 00:36:13.118598 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:13.118528    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" path="/var/lib/kubelet/pods/4a696228-3344-41d5-8f66-6e0f12e86723/volumes"
Oct 31 00:36:13.119021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:13.119006    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" path="/var/lib/kubelet/pods/9b8a74a7-cbc9-46f4-a670-c286729d4f27/volumes"
Oct 31 00:36:13.119431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:13.119417    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" path="/var/lib/kubelet/pods/b169c664-7798-4ef4-bdfa-efbadc4ccfd4/volumes"
Oct 31 00:36:13.120123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:13.120109    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" path="/var/lib/kubelet/pods/cbfa66e8-6d8a-4814-81c1-6e2d7c685d84/volumes"
Oct 31 00:36:14.763459 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:14.763370    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:36:16.423408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.422934    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"]
Oct 31 00:36:16.423935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423862    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy"
Oct 31 00:36:16.423935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423882    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy"
Oct 31 00:36:16.423935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423897    4478 cpu_manager.go:414] "containerMap: removing container" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy"
Oct 31 00:36:16.423935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423905    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423940    4478 cpu_manager.go:414] "containerMap: removing container" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423949    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423985    4478 cpu_manager.go:414] "containerMap: removing container" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.423992    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.424138    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a696228-3344-41d5-8f66-6e0f12e86723" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.424154    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8a74a7-cbc9-46f4-a670-c286729d4f27" containerName="istio-proxy"
Oct 31 00:36:16.424168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.424170    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b169c664-7798-4ef4-bdfa-efbadc4ccfd4" containerName="istio-proxy"
Oct 31 00:36:16.424540 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.424178    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfa66e8-6d8a-4814-81c1-6e2d7c685d84" containerName="istio-proxy"
Oct 31 00:36:16.505416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.503826    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"]
Oct 31 00:36:16.505416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.503886    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"]
Oct 31 00:36:16.505416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.504088    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" podUID="03ad984f-11e3-40c8-be50-a874b1b755e6" containerName="maas-api" containerID="cri-o://c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3" gracePeriod=30
Oct 31 00:36:16.505416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.504295    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.574620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.572756    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-752be8\""
Oct 31 00:36:16.574620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.573279    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-752be8-dockercfg-vs46p\""
Oct 31 00:36:16.574620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.574077    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tmv4\" (UniqueName: \"kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4\") pod \"payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.574620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.574259    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.686239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.678701    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.686239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.678820    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tmv4\" (UniqueName: \"kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4\") pod \"payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.696966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.693318    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume\") pod \"payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.740557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.740524    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tmv4\" (UniqueName: \"kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4\") pod \"payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") " pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:16.893406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.858473    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:36:16.893406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.858539    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:36:16.956369 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.954575    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:36:16.979627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:16.978662    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:17.088433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.087605    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s\" (UniqueName: \"kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s\") pod \"03ad984f-11e3-40c8-be50-a874b1b755e6\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") "
Oct 31 00:36:17.088433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.087666    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca\") pod \"03ad984f-11e3-40c8-be50-a874b1b755e6\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") "
Oct 31 00:36:17.088433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.087701    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\") pod \"03ad984f-11e3-40c8-be50-a874b1b755e6\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") "
Oct 31 00:36:17.088433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.087927    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs\") pod \"03ad984f-11e3-40c8-be50-a874b1b755e6\" (UID: \"03ad984f-11e3-40c8-be50-a874b1b755e6\") "
Oct 31 00:36:17.088785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.088508    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca" pod "03ad984f-11e3-40c8-be50-a874b1b755e6" (UID: "03ad984f-11e3-40c8-be50-a874b1b755e6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:17.090777 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.090748    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs" pod "03ad984f-11e3-40c8-be50-a874b1b755e6" (UID: "03ad984f-11e3-40c8-be50-a874b1b755e6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:17.091186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.091161    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s" pod "03ad984f-11e3-40c8-be50-a874b1b755e6" (UID: "03ad984f-11e3-40c8-be50-a874b1b755e6"). InnerVolumeSpecName "kube-api-access-qws9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:17.091274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.091252    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls" pod "03ad984f-11e3-40c8-be50-a874b1b755e6" (UID: "03ad984f-11e3-40c8-be50-a874b1b755e6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:17.190784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.190757    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.190939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.190807    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qws9s\" (UniqueName: \"kubernetes.io/projected/03ad984f-11e3-40c8-be50-a874b1b755e6-kube-api-access-qws9s\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.190939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.190818    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ad984f-11e3-40c8-be50-a874b1b755e6-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.190939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.190829    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ad984f-11e3-40c8-be50-a874b1b755e6-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.359970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.359502    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"]
Oct 31 00:36:17.360944 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:36:17.360917    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod624d5193_dbd6_4d88_94f9_3154f5f2c7cb.slice/crio-c1b3ea69b264439d151d5a801c8cf50cbb792d28c185878e49171c702913f3ac WatchSource:0}: Error finding container c1b3ea69b264439d151d5a801c8cf50cbb792d28c185878e49171c702913f3ac: Status 404 returned error can't find the container with id c1b3ea69b264439d151d5a801c8cf50cbb792d28c185878e49171c702913f3ac
Oct 31 00:36:17.518067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.518038    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:36:17.599555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.599526    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"]
Oct 31 00:36:17.599772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.599750    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" podUID="2484b7fe-bf88-477e-8b56-4a82a8b3b406" containerName="maas-api" containerID="cri-o://833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9" gracePeriod=30
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612169    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612251    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612288    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612422    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612423    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612499    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612536    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612543    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612611    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612675    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612716    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.612755    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6\") pod \"aa1075b1-41b0-4857-8701-657c5332c744\" (UID: \"aa1075b1-41b0-4857-8701-657c5332c744\") "
Oct 31 00:36:17.613334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.613090    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:17.614103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.613518    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.614103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.613542    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.614103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.613556    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aa1075b1-41b0-4857-8701-657c5332c744-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.614103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.613565    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.614103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.613921    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:17.618512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.618484    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s" event={"ID":"624d5193-dbd6-4d88-94f9-3154f5f2c7cb","Type":"ContainerStarted","Data":"eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6"}
Oct 31 00:36:17.618637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.618564    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:17.618637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.618590    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s" event={"ID":"624d5193-dbd6-4d88-94f9-3154f5f2c7cb","Type":"ContainerStarted","Data":"c1b3ea69b264439d151d5a801c8cf50cbb792d28c185878e49171c702913f3ac"}
Oct 31 00:36:17.620144 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.620119    4478 generic.go:356] "Generic (PLEG): container finished" podID="aa1075b1-41b0-4857-8701-657c5332c744" containerID="05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88" exitCode=0
Oct 31 00:36:17.620256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.620193    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" event={"ID":"aa1075b1-41b0-4857-8701-657c5332c744","Type":"ContainerDied","Data":"05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88"}
Oct 31 00:36:17.620256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.620239    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg" event={"ID":"aa1075b1-41b0-4857-8701-657c5332c744","Type":"ContainerDied","Data":"d65eabea9c754a5f929067b3c580fa4c177b7593f58fdd09f7da60e823b6c5de"}
Oct 31 00:36:17.620372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.620266    4478 scope.go:122] "RemoveContainer" containerID="05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88"
Oct 31 00:36:17.620447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.620409    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"
Oct 31 00:36:17.621874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.621852    4478 generic.go:356] "Generic (PLEG): container finished" podID="03ad984f-11e3-40c8-be50-a874b1b755e6" containerID="c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3" exitCode=0
Oct 31 00:36:17.621971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.621896    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"
Oct 31 00:36:17.621971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.621925    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" event={"ID":"03ad984f-11e3-40c8-be50-a874b1b755e6","Type":"ContainerDied","Data":"c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3"}
Oct 31 00:36:17.621971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.621965    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd" event={"ID":"03ad984f-11e3-40c8-be50-a874b1b755e6","Type":"ContainerDied","Data":"f3b50538bd63637203756ab8c65319134a2596d040344105d834ac79757b3ad1"}
Oct 31 00:36:17.623781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.623755    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:17.633033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.632910    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:17.633033 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.632997    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "kube-api-access-fjsx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:17.633498 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.633471    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo" pod "aa1075b1-41b0-4857-8701-657c5332c744" (UID: "aa1075b1-41b0-4857-8701-657c5332c744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:36:17.711906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.711785    4478 scope.go:122] "RemoveContainer" containerID="05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88"
Oct 31 00:36:17.712226 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:17.712179    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88\": container with ID starting with 05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88 not found: ID does not exist" containerID="05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88"
Oct 31 00:36:17.712271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.712244    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88"} err="failed to get container status \"05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88\": rpc error: code = NotFound desc = could not find container \"05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88\": container with ID starting with 05f1d505a01b3b71a383f9e26ab29cd44b730e6b34dca854b869eb738bfbba88 not found: ID does not exist"
Oct 31 00:36:17.712316 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.712273    4478 scope.go:122] "RemoveContainer" containerID="c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3"
Oct 31 00:36:17.715485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.715459    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aa1075b1-41b0-4857-8701-657c5332c744-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.715485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.715490    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.715655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.715507    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.715655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.715524    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aa1075b1-41b0-4857-8701-657c5332c744-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.715655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.715540    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjsx6\" (UniqueName: \"kubernetes.io/projected/aa1075b1-41b0-4857-8701-657c5332c744-kube-api-access-fjsx6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:17.725971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.725952    4478 scope.go:122] "RemoveContainer" containerID="c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3"
Oct 31 00:36:17.726262 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:17.726195    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3\": container with ID starting with c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3 not found: ID does not exist" containerID="c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3"
Oct 31 00:36:17.726262 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.726221    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3"} err="failed to get container status \"c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3\": rpc error: code = NotFound desc = could not find container \"c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3\": container with ID starting with c4e13d3c535ed0db037d437c317d0386a301916e8b9513b49e27691fe4beaee3 not found: ID does not exist"
Oct 31 00:36:17.929134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.910985    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s" podStartSLOduration=1.9109612280000001 podStartE2EDuration="1.910961228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:17.874323591 +0000 UTC m=+2010.890988328" watchObservedRunningTime="2026-10-31 00:36:17.910961228 +0000 UTC m=+2010.927625942"
Oct 31 00:36:17.979272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:17.979238    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:36:18.084544 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.084478    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca\") pod \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") "
Oct 31 00:36:18.084684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.084604    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf\" (UniqueName: \"kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf\") pod \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") "
Oct 31 00:36:18.084684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.084627    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs\") pod \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") "
Oct 31 00:36:18.084684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.084677    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\") pod \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\" (UID: \"2484b7fe-bf88-477e-8b56-4a82a8b3b406\") "
Oct 31 00:36:18.084857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.084814    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca" pod "2484b7fe-bf88-477e-8b56-4a82a8b3b406" (UID: "2484b7fe-bf88-477e-8b56-4a82a8b3b406"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:18.085332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.085264    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2484b7fe-bf88-477e-8b56-4a82a8b3b406-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:18.087067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.087036    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs" pod "2484b7fe-bf88-477e-8b56-4a82a8b3b406" (UID: "2484b7fe-bf88-477e-8b56-4a82a8b3b406"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:18.087165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.087086    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf" pod "2484b7fe-bf88-477e-8b56-4a82a8b3b406" (UID: "2484b7fe-bf88-477e-8b56-4a82a8b3b406"). InnerVolumeSpecName "kube-api-access-97vlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:18.087165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.087106    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls" pod "2484b7fe-bf88-477e-8b56-4a82a8b3b406" (UID: "2484b7fe-bf88-477e-8b56-4a82a8b3b406"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:36:18.197620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.197575    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"]
Oct 31 00:36:18.200413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.200368    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:18.200652 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.200611    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97vlf\" (UniqueName: \"kubernetes.io/projected/2484b7fe-bf88-477e-8b56-4a82a8b3b406-kube-api-access-97vlf\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:18.200652 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.200650    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2484b7fe-bf88-477e-8b56-4a82a8b3b406-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:18.208687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.208659    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-wrcjg"]
Oct 31 00:36:18.400898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.400806    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"]
Oct 31 00:36:18.429756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.429711    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3e7872-8647bdb69b-xggkd"]
Oct 31 00:36:18.660510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.660410    4478 generic.go:356] "Generic (PLEG): container finished" podID="2484b7fe-bf88-477e-8b56-4a82a8b3b406" containerID="833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9" exitCode=0
Oct 31 00:36:18.661256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.661213    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" event={"ID":"2484b7fe-bf88-477e-8b56-4a82a8b3b406","Type":"ContainerDied","Data":"833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9"}
Oct 31 00:36:18.662396 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.662352    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc" event={"ID":"2484b7fe-bf88-477e-8b56-4a82a8b3b406","Type":"ContainerDied","Data":"a16849a3671f934e66a8ceb51f558f7d0a921e09f3731e41e44ae7cf59d66417"}
Oct 31 00:36:18.662563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.662547    4478 scope.go:122] "RemoveContainer" containerID="833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9"
Oct 31 00:36:18.662810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.662796    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"
Oct 31 00:36:18.678844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.678819    4478 scope.go:122] "RemoveContainer" containerID="833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9"
Oct 31 00:36:18.679126 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:18.679103    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9\": container with ID starting with 833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9 not found: ID does not exist" containerID="833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9"
Oct 31 00:36:18.679250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.679224    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9"} err="failed to get container status \"833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9\": rpc error: code = NotFound desc = could not find container \"833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9\": container with ID starting with 833770f0c98148a3e828699b1831747d149694841e6c3f85e90524bc98976ea9 not found: ID does not exist"
Oct 31 00:36:18.827939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.827898    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"]
Oct 31 00:36:18.830184 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:18.830157    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-2a59a7-7f55ffb97d-h4sgc"]
Oct 31 00:36:19.117811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:19.117781    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ad984f-11e3-40c8-be50-a874b1b755e6" path="/var/lib/kubelet/pods/03ad984f-11e3-40c8-be50-a874b1b755e6/volumes"
Oct 31 00:36:19.118255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:19.118239    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2484b7fe-bf88-477e-8b56-4a82a8b3b406" path="/var/lib/kubelet/pods/2484b7fe-bf88-477e-8b56-4a82a8b3b406/volumes"
Oct 31 00:36:19.118713 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:19.118698    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa1075b1-41b0-4857-8701-657c5332c744" path="/var/lib/kubelet/pods/aa1075b1-41b0-4857-8701-657c5332c744/volumes"
Oct 31 00:36:27.158942 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:27.158909    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"]
Oct 31 00:36:27.159414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:27.159114    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy" containerID="cri-o://18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde" gracePeriod=30
Oct 31 00:36:28.349361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.349334    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 00:36:28.641921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.641824    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 00:36:28.672801 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.672778    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:36:28.851229 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851194    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"]
Oct 31 00:36:28.851796 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851776    4478 cpu_manager.go:414] "containerMap: removing container" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy"
Oct 31 00:36:28.851796 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851795    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851832    4478 cpu_manager.go:414] "containerMap: removing container" podUID="03ad984f-11e3-40c8-be50-a874b1b755e6" containerName="maas-api"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851838    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ad984f-11e3-40c8-be50-a874b1b755e6" containerName="maas-api"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851845    4478 cpu_manager.go:414] "containerMap: removing container" podUID="2484b7fe-bf88-477e-8b56-4a82a8b3b406" containerName="maas-api"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851850    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2484b7fe-bf88-477e-8b56-4a82a8b3b406" containerName="maas-api"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851926    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ad984f-11e3-40c8-be50-a874b1b755e6" containerName="maas-api"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851933    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="2484b7fe-bf88-477e-8b56-4a82a8b3b406" containerName="maas-api"
Oct 31 00:36:28.851968 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.851940    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa1075b1-41b0-4857-8701-657c5332c744" containerName="istio-proxy"
Oct 31 00:36:28.856697 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.856673    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903117    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903201    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903236    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903261    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903286    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46h2q\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903336    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903361    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903539    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.903671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.903585    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:28.908779 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.908747    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-97fc39-openshift-default-dockercfg-k65db\""
Oct 31 00:36:28.918756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:28.918736    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"]
Oct 31 00:36:29.025213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.025174    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.025506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.025481    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.025793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.025776    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.025925 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.025911    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.026077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.026059    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.026220 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.026201    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.026347 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.026329    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.026494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.026478    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.026600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.026586    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46h2q\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.027470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.027441    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.052404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.028223    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.053034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.053005    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.053754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.053727    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.054136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.054115    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.084415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.073712    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.085251 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.085219    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.099600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.099556    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46h2q\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.131971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.128889    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token\") pod \"e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") " pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.170747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.170654    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:29.609003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.608978    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"]
Oct 31 00:36:29.610832 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:36:29.610807    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb36bef42_9e41_4da7_a998_103b93a9e39c.slice/crio-6779c5260fee53a38bc76aae0424ed9ea827cf48675811ba6ed578d0def96a94 WatchSource:0}: Error finding container 6779c5260fee53a38bc76aae0424ed9ea827cf48675811ba6ed578d0def96a94: Status 404 returned error can't find the container with id 6779c5260fee53a38bc76aae0424ed9ea827cf48675811ba6ed578d0def96a94
Oct 31 00:36:29.613312 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.613285    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:36:29.613412 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.613363    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:36:29.613471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.613432    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:36:29.743704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.743673    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" event={"ID":"b36bef42-9e41-4da7-a998-103b93a9e39c","Type":"ContainerStarted","Data":"263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"}
Oct 31 00:36:29.743891 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.743733    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" event={"ID":"b36bef42-9e41-4da7-a998-103b93a9e39c","Type":"ContainerStarted","Data":"6779c5260fee53a38bc76aae0424ed9ea827cf48675811ba6ed578d0def96a94"}
Oct 31 00:36:29.865244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:29.865152    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podStartSLOduration=1.865137883 podStartE2EDuration="1.865137883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:29.862898172 +0000 UTC m=+2022.879562887" watchObservedRunningTime="2026-10-31 00:36:29.865137883 +0000 UTC m=+2022.881802595"
Oct 31 00:36:30.170905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:30.170822    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:30.177572 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:30.177538    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:30.749453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:30.749424    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:30.750322 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:30.750302    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:36:32.418321 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.418295    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:36:32.575103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575073    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575114    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575141    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575217    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575249    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575285    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575329    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575352    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:32.575543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575457    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575572    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket\") pod \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\" (UID: \"f4564be4-23f3-4588-a910-7f3f8d1ec13e\") "
Oct 31 00:36:32.575747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575618    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:32.575747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575531    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:32.575747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575633    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:32.575922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.575895    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:32.576213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.576074    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.576213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.576102    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.576213 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.576118    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.576493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.576195    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.576493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.576305    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.578085 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.578058    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:36:32.578197 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.578105    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:32.578460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.578437    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:36:32.578552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.578518    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz" pod "f4564be4-23f3-4588-a910-7f3f8d1ec13e" (UID: "f4564be4-23f3-4588-a910-7f3f8d1ec13e"). InnerVolumeSpecName "kube-api-access-9zclz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:32.677872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.677837    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.677872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.677867    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.678061 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.677879    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zclz\" (UniqueName: \"kubernetes.io/projected/f4564be4-23f3-4588-a910-7f3f8d1ec13e-kube-api-access-9zclz\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.678061 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.677889    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4564be4-23f3-4588-a910-7f3f8d1ec13e-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:32.763060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.763027    4478 generic.go:356] "Generic (PLEG): container finished" podID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerID="18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde" exitCode=0
Oct 31 00:36:32.763499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.763097    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"
Oct 31 00:36:32.763499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.763119    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" event={"ID":"f4564be4-23f3-4588-a910-7f3f8d1ec13e","Type":"ContainerDied","Data":"18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde"}
Oct 31 00:36:32.763499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.763194    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l" event={"ID":"f4564be4-23f3-4588-a910-7f3f8d1ec13e","Type":"ContainerDied","Data":"61370d95557fff179972f400be46ec49e6b629e898f61fae499ccb2b4f01423c"}
Oct 31 00:36:32.763499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.763215    4478 scope.go:122] "RemoveContainer" containerID="18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde"
Oct 31 00:36:32.773296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.773280    4478 scope.go:122] "RemoveContainer" containerID="5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"
Oct 31 00:36:32.782683 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.782666    4478 scope.go:122] "RemoveContainer" containerID="18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde"
Oct 31 00:36:32.782948 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:32.782929    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde\": container with ID starting with 18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde not found: ID does not exist" containerID="18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde"
Oct 31 00:36:32.783001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.782956    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde"} err="failed to get container status \"18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde\": rpc error: code = NotFound desc = could not find container \"18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde\": container with ID starting with 18fcbde93efb1f0715c6ede8a38131b00f6f20dbc50dfc68eeddb9ef2059efde not found: ID does not exist"
Oct 31 00:36:32.783001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.782973    4478 scope.go:122] "RemoveContainer" containerID="5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"
Oct 31 00:36:32.783187 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:32.783173    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37\": container with ID starting with 5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37 not found: ID does not exist" containerID="5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"
Oct 31 00:36:32.783229 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.783192    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37"} err="failed to get container status \"5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37\": rpc error: code = NotFound desc = could not find container \"5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37\": container with ID starting with 5b8e96b2ceee1750c985ae02bf9a2baa75a89df229a42c3aa924cb5171ec5a37 not found: ID does not exist"
Oct 31 00:36:32.934357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.934259    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"]
Oct 31 00:36:32.936788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:32.936763    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-2a59a7-openshift-default-56675cf64f-ktm7l"]
Oct 31 00:36:33.119563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:33.119532    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" path="/var/lib/kubelet/pods/f4564be4-23f3-4588-a910-7f3f8d1ec13e/volumes"
Oct 31 00:36:34.929752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.929724    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"]
Oct 31 00:36:34.930454 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.930434    4478 cpu_manager.go:414] "containerMap: removing container" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy"
Oct 31 00:36:34.930573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.930455    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy"
Oct 31 00:36:34.930628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.930607    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy"
Oct 31 00:36:34.930628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.930624    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy"
Oct 31 00:36:34.935449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.935430    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:34.987354 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:34.987329    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-f6a8a5-serving-cert\""
Oct 31 00:36:35.040789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.040761    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"]
Oct 31 00:36:35.102861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.102819    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.103039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.102873    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.103039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.102963    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.103039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.103001    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xqmh\" (UniqueName: \"kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.225871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.205293    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.225871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.205350    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.225871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.205398    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.225871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.205442    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xqmh\" (UniqueName: \"kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.225871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.207764    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.229754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.229718    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.240018 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.239947    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.269085 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.269058    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xqmh\" (UniqueName: \"kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh\") pod \"maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.550415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.550314    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:35.932743 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.932711    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"]
Oct 31 00:36:35.933829 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:36:35.933805    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab6d036_7c66_488f_a5e3_05f3bc47bab3.slice/crio-f8dbbe6ed582d811215cd77e51cfcecf23d554618877791181713b04e88ecc6b WatchSource:0}: Error finding container f8dbbe6ed582d811215cd77e51cfcecf23d554618877791181713b04e88ecc6b: Status 404 returned error can't find the container with id f8dbbe6ed582d811215cd77e51cfcecf23d554618877791181713b04e88ecc6b
Oct 31 00:36:35.935451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:35.935431    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:36:36.791192 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:36.791156    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc" event={"ID":"1ab6d036-7c66-488f-a5e3-05f3bc47bab3","Type":"ContainerStarted","Data":"f8dbbe6ed582d811215cd77e51cfcecf23d554618877791181713b04e88ecc6b"}
Oct 31 00:36:37.681514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:37.681479    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"]
Oct 31 00:36:37.774333 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:37.774299    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"]
Oct 31 00:36:37.799267 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:37.799233    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc" event={"ID":"1ab6d036-7c66-488f-a5e3-05f3bc47bab3","Type":"ContainerStarted","Data":"f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7"}
Oct 31 00:36:37.799888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:37.799872    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:38.029933 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:38.029836    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc" podStartSLOduration=3.091746592 podStartE2EDuration="4.029821516s" totalImagesPullingTime="938.074924ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:38.02877999 +0000 UTC m=+2031.045444699" watchObservedRunningTime="2026-10-31 00:36:38.029821516 +0000 UTC m=+2031.046486228"
Oct 31 00:36:41.957156 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:41.957125    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"]
Oct 31 00:36:41.957679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:41.957664    4478 cpu_manager.go:414] "containerMap: removing container" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy"
Oct 31 00:36:41.957679 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:41.957679    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4564be4-23f3-4588-a910-7f3f8d1ec13e" containerName="istio-proxy"
Oct 31 00:36:41.960989 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:41.960972    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.006745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.006717    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.006898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.006883    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crc9v\" (UniqueName: \"kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.082882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.082777    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"]
Oct 31 00:36:42.083232 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.083196    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97fc39-dockercfg-td9t8\""
Oct 31 00:36:42.083775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.083743    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97fc39\""
Oct 31 00:36:42.115567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.115527    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.117643 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.117495    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.118157 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.118134    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crc9v\" (UniqueName: \"kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.187405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.186756    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crc9v\" (UniqueName: \"kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v\") pod \"payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.274059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.273986    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:42.650357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.650336    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"]
Oct 31 00:36:42.651885 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:36:42.651859    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae22bad8_5066_4c9e_be89_b41463ac4807.slice/crio-5a7bf261e2a532dec9934ba3764b16b32bff90bcb8b9a523dff97447ba89180a WatchSource:0}: Error finding container 5a7bf261e2a532dec9934ba3764b16b32bff90bcb8b9a523dff97447ba89180a: Status 404 returned error can't find the container with id 5a7bf261e2a532dec9934ba3764b16b32bff90bcb8b9a523dff97447ba89180a
Oct 31 00:36:42.681882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.681853    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g" podUID="db4ef832-f1b5-470b-828f-ebe2c31df5d9" containerName="payload-processing" containerID="cri-o://f55e07baf1eb445466ec18ef80d46cf2acb240a5fb68aba711c4f7029f10cefe" gracePeriod=25
Oct 31 00:36:42.775302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.775267    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg" podUID="332bac2a-f38e-4368-8534-b889aad36e42" containerName="payload-pre-processing" containerID="cri-o://b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502" gracePeriod=25
Oct 31 00:36:42.872483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.872443    4478 generic.go:356] "Generic (PLEG): container finished" podID="db4ef832-f1b5-470b-828f-ebe2c31df5d9" containerID="f55e07baf1eb445466ec18ef80d46cf2acb240a5fb68aba711c4f7029f10cefe" exitCode=0
Oct 31 00:36:42.872635 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.872525    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g" event={"ID":"db4ef832-f1b5-470b-828f-ebe2c31df5d9","Type":"ContainerDied","Data":"f55e07baf1eb445466ec18ef80d46cf2acb240a5fb68aba711c4f7029f10cefe"}
Oct 31 00:36:42.874311 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:42.874288    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt" event={"ID":"ae22bad8-5066-4c9e-be89-b41463ac4807","Type":"ContainerStarted","Data":"5a7bf261e2a532dec9934ba3764b16b32bff90bcb8b9a523dff97447ba89180a"}
Oct 31 00:36:43.024862 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.024837    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:36:43.089703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.089679    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:36:43.131669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.131637    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume\") pod \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") "
Oct 31 00:36:43.131669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.131681    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv\" (UniqueName: \"kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv\") pod \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\" (UID: \"db4ef832-f1b5-470b-828f-ebe2c31df5d9\") "
Oct 31 00:36:43.131910 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.131726    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n\" (UniqueName: \"kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n\") pod \"332bac2a-f38e-4368-8534-b889aad36e42\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") "
Oct 31 00:36:43.131910 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.131869    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume\") pod \"332bac2a-f38e-4368-8534-b889aad36e42\" (UID: \"332bac2a-f38e-4368-8534-b889aad36e42\") "
Oct 31 00:36:43.132022 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.131987    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume" pod "db4ef832-f1b5-470b-828f-ebe2c31df5d9" (UID: "db4ef832-f1b5-470b-828f-ebe2c31df5d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:43.132340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.132318    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db4ef832-f1b5-470b-828f-ebe2c31df5d9-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:43.132531 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.132482    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume" pod "332bac2a-f38e-4368-8534-b889aad36e42" (UID: "332bac2a-f38e-4368-8534-b889aad36e42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:36:43.134042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.134017    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv" pod "db4ef832-f1b5-470b-828f-ebe2c31df5d9" (UID: "db4ef832-f1b5-470b-828f-ebe2c31df5d9"). InnerVolumeSpecName "kube-api-access-6hczv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:43.134130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.134039    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n" pod "332bac2a-f38e-4368-8534-b889aad36e42" (UID: "332bac2a-f38e-4368-8534-b889aad36e42"). InnerVolumeSpecName "kube-api-access-j7x5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:36:43.233331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.233303    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/332bac2a-f38e-4368-8534-b889aad36e42-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:43.233331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.233331    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hczv\" (UniqueName: \"kubernetes.io/projected/db4ef832-f1b5-470b-828f-ebe2c31df5d9-kube-api-access-6hczv\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:43.233537 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.233342    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7x5n\" (UniqueName: \"kubernetes.io/projected/332bac2a-f38e-4368-8534-b889aad36e42-kube-api-access-j7x5n\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:36:43.881908 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.881872    4478 generic.go:356] "Generic (PLEG): container finished" podID="332bac2a-f38e-4368-8534-b889aad36e42" containerID="b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502" exitCode=0
Oct 31 00:36:43.882068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.881913    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"
Oct 31 00:36:43.882068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.881957    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg" event={"ID":"332bac2a-f38e-4368-8534-b889aad36e42","Type":"ContainerDied","Data":"b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502"}
Oct 31 00:36:43.882068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.882012    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg" event={"ID":"332bac2a-f38e-4368-8534-b889aad36e42","Type":"ContainerDied","Data":"0965ab34686546199d71f7c02bb868f8d5eb3bba56799ab9420b66c3edbc640e"}
Oct 31 00:36:43.882068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.882036    4478 scope.go:122] "RemoveContainer" containerID="b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502"
Oct 31 00:36:43.884501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.884132    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g" event={"ID":"db4ef832-f1b5-470b-828f-ebe2c31df5d9","Type":"ContainerDied","Data":"70fa0516cdbc1d1ae4eeabd119f784f89e5b8836b157be2cd33d719001fd68d2"}
Oct 31 00:36:43.884501 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.884135    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"
Oct 31 00:36:43.889147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.889120    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt" event={"ID":"ae22bad8-5066-4c9e-be89-b41463ac4807","Type":"ContainerStarted","Data":"463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a"}
Oct 31 00:36:43.889284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.889265    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:43.893266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.893248    4478 scope.go:122] "RemoveContainer" containerID="b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502"
Oct 31 00:36:43.893630 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:43.893600    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502\": container with ID starting with b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502 not found: ID does not exist" containerID="b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502"
Oct 31 00:36:43.893739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.893639    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502"} err="failed to get container status \"b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502\": rpc error: code = NotFound desc = could not find container \"b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502\": container with ID starting with b27d152863c8594d0f23394f1b82a55870bb0d7af51c2446550c8f9e9c879502 not found: ID does not exist"
Oct 31 00:36:43.893739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:43.893664    4478 scope.go:122] "RemoveContainer" containerID="f55e07baf1eb445466ec18ef80d46cf2acb240a5fb68aba711c4f7029f10cefe"
Oct 31 00:36:44.056209 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:44.056175    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"]
Oct 31 00:36:44.104693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:44.104665    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2cda64-8599f5c5fc-dl84g"]
Oct 31 00:36:44.164208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:44.164121    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"]
Oct 31 00:36:44.169021 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:44.168994    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2cda64-5c4c7bc8d7-c2sjg"]
Oct 31 00:36:44.282521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:44.282466    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt" podStartSLOduration=3.282450601 podStartE2EDuration="3.282450601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:36:44.281536806 +0000 UTC m=+2037.298201536" watchObservedRunningTime="2026-10-31 00:36:44.282450601 +0000 UTC m=+2037.299115313"
Oct 31 00:36:44.758855 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:36:44.758807    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:36:44.815257 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:44.815174    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:36:45.117547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:45.117456    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="332bac2a-f38e-4368-8534-b889aad36e42" path="/var/lib/kubelet/pods/332bac2a-f38e-4368-8534-b889aad36e42/volumes"
Oct 31 00:36:45.118056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:45.118034    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4ef832-f1b5-470b-828f-ebe2c31df5d9" path="/var/lib/kubelet/pods/db4ef832-f1b5-470b-828f-ebe2c31df5d9/volumes"
Oct 31 00:36:54.900289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:54.900265    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:36:59.908423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.908364    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"]
Oct 31 00:36:59.910127 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.910098    4478 cpu_manager.go:414] "containerMap: removing container" podUID="db4ef832-f1b5-470b-828f-ebe2c31df5d9" containerName="payload-processing"
Oct 31 00:36:59.910290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.910275    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4ef832-f1b5-470b-828f-ebe2c31df5d9" containerName="payload-processing"
Oct 31 00:36:59.910409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.910375    4478 cpu_manager.go:414] "containerMap: removing container" podUID="332bac2a-f38e-4368-8534-b889aad36e42" containerName="payload-pre-processing"
Oct 31 00:36:59.910506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.910494    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="332bac2a-f38e-4368-8534-b889aad36e42" containerName="payload-pre-processing"
Oct 31 00:36:59.910914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.910894    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4ef832-f1b5-470b-828f-ebe2c31df5d9" containerName="payload-processing"
Oct 31 00:36:59.911047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.911036    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="332bac2a-f38e-4368-8534-b889aad36e42" containerName="payload-pre-processing"
Oct 31 00:36:59.917812 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.917791    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:36:59.953254 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.953227    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:36:59.953451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.953274    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rw6nv\" (UniqueName: \"kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:36:59.953451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.953313    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:36:59.953600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.953460    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:36:59.953600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.953487    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:36:59.953600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:36:59.953505    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.016697 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.016662    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bba5ed\"/\"kube-root-ca.crt\""
Oct 31 00:37:00.019689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.019652    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bba5ed\"/\"ipp-route-bba5ed-kserve-self-signed-certs\""
Oct 31 00:37:00.019689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.019661    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-bba5ed\"/\"default-dockercfg-f64pw\""
Oct 31 00:37:00.019886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.019859    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-bba5ed\"/\"openshift-service-ca.crt\""
Oct 31 00:37:00.022342 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.022323    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"]
Oct 31 00:37:00.067477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.059246    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.067821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.067789    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.071487 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.071458    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.102915 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.102881    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.103245 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.103214    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rw6nv\" (UniqueName: \"kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.103438 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.074820    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.103624 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.103417    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.103765 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.103740    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.103925 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.068109    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.104432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.097164    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.125564 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.125533    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.174013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.173980    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rw6nv\" (UniqueName: \"kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv\") pod \"ipp-route-bba5ed-kserve-775c559cd5-62rz2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") " pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.230848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.230807    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:00.613439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.613415    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"]
Oct 31 00:37:00.615561 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:37:00.615531    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0793873_7f1d_42b3_8343_efd7e744bad2.slice/crio-b9cb32d0c6e1aed91ed0383c9c73d4d9c64f28cc3980d76829b3973e5f713c7e WatchSource:0}: Error finding container b9cb32d0c6e1aed91ed0383c9c73d4d9c64f28cc3980d76829b3973e5f713c7e: Status 404 returned error can't find the container with id b9cb32d0c6e1aed91ed0383c9c73d4d9c64f28cc3980d76829b3973e5f713c7e
Oct 31 00:37:00.985963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.985927    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2" event={"ID":"c0793873-7f1d-42b3-8343-efd7e744bad2","Type":"ContainerStarted","Data":"e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806"}
Oct 31 00:37:00.986392 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:00.985976    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2" event={"ID":"c0793873-7f1d-42b3-8343-efd7e744bad2","Type":"ContainerStarted","Data":"b9cb32d0c6e1aed91ed0383c9c73d4d9c64f28cc3980d76829b3973e5f713c7e"}
Oct 31 00:37:01.104294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:01.104231    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2" podStartSLOduration=2.104212133 podStartE2EDuration="2.104212133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:01.102503694 +0000 UTC m=+2054.119168427" watchObservedRunningTime="2026-10-31 00:37:01.104212133 +0000 UTC m=+2054.120876849"
Oct 31 00:37:01.241774 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:01.241695    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:10.231317 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:10.231280    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:10.245023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:10.244998    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:11.060176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:11.060145    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:37:14.762716 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:37:14.762675    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:37:27.568798 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.568759    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"]
Oct 31 00:37:27.573849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.573818    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.600028 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.599999    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"]
Oct 31 00:37:27.617681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.617651    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.617821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.617703    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.617821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.617792    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxk4t\" (UniqueName: \"kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.617969 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.617954    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.663003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.662871    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-97fc39-serving-cert\""
Oct 31 00:37:27.719204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.719166    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.719367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.719252    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.719367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.719284    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.719367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.719307    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxk4t\" (UniqueName: \"kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.719367 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:37:27.719329    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-serving-cert: secret "maas-api-e2e-ipp-a-97fc39-serving-cert" not found
Oct 31 00:37:27.719580 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:37:27.719418    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls podName:30ab8f75-f762-4a19-aedc-51919c61766a nodeName:}" failed. No retries permitted until 2026-10-31 00:37:28.219396857 +0000 UTC m=+2081.236061563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls") pod "maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" (UID: "30ab8f75-f762-4a19-aedc-51919c61766a") : secret "maas-api-e2e-ipp-a-97fc39-serving-cert" not found
Oct 31 00:37:27.720019 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.720003    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.722031 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.722009    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:27.789150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:27.789113    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxk4t\" (UniqueName: \"kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:28.264438 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:28.264404    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:28.267042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:28.267019    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-97fc39-647d85754d-d552c\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:28.492059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:28.492030    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:28.895204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:28.895172    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"]
Oct 31 00:37:28.896196 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:37:28.896171    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30ab8f75_f762_4a19_aedc_51919c61766a.slice/crio-099ce724e789cdcc833485bcd949f1f86cd981696cfe294bc4f330aca36308d7 WatchSource:0}: Error finding container 099ce724e789cdcc833485bcd949f1f86cd981696cfe294bc4f330aca36308d7: Status 404 returned error can't find the container with id 099ce724e789cdcc833485bcd949f1f86cd981696cfe294bc4f330aca36308d7
Oct 31 00:37:29.221706 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:29.221668    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" event={"ID":"30ab8f75-f762-4a19-aedc-51919c61766a","Type":"ContainerStarted","Data":"099ce724e789cdcc833485bcd949f1f86cd981696cfe294bc4f330aca36308d7"}
Oct 31 00:37:30.229326 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:30.229286    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" event={"ID":"30ab8f75-f762-4a19-aedc-51919c61766a","Type":"ContainerStarted","Data":"77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9"}
Oct 31 00:37:30.229842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:30.229449    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:30.349700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:30.349639    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" podStartSLOduration=2.542581089 podStartE2EDuration="3.349624108s" totalImagesPullingTime="807.043019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:37:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:37:30.346697062 +0000 UTC m=+2083.363361774" watchObservedRunningTime="2026-10-31 00:37:30.349624108 +0000 UTC m=+2083.366288819"
Oct 31 00:37:32.243016 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:32.242982    4478 generic.go:356] "Generic (PLEG): container finished" podID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerID="edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c" exitCode=0
Oct 31 00:37:32.243450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:32.243063    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" event={"ID":"079636e4-f2e7-4c52-8bc7-880d0149f24c","Type":"ContainerDied","Data":"edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"}
Oct 31 00:37:32.243552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:32.243533    4478 scope.go:122] "RemoveContainer" containerID="edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"
Oct 31 00:37:32.245130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:32.245106    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:37:32.245196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:32.245172    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:37:32.245236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:32.245203    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:37:33.268867 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:33.268831    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" event={"ID":"079636e4-f2e7-4c52-8bc7-880d0149f24c","Type":"ContainerStarted","Data":"03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8"}
Oct 31 00:37:33.296957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:33.296927    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:37:33.297123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:33.297030    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:37:33.297123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:33.297074    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:37:34.297488 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:34.297451    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:37:34.297875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:34.297507    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:37:35.297691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:35.297657    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:37:35.298097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:35.297708    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:37:36.256854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:36.256826    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:37:36.297036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:36.297003    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:37:36.297180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:36.297052    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:37:37.297698 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:37.297670    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:37:37.298083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:37.297725    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:37:38.297513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:38.297470    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Oct 31 00:37:38.297698 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:38.297539    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Oct 31 00:37:39.301342 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:39.301312    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:37:39.301834 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:39.301809    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:37:39.302603 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:39.302582    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:37:44.758892 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:37:44.758849    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:37:52.865269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:52.865233    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"]
Oct 31 00:37:52.865800 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:52.865493    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" containerID="cri-o://8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd" gracePeriod=30
Oct 31 00:37:55.649410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:55.648442    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:37:55.649410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:55.648525    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:37:58.047208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.047168    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"]
Oct 31 00:37:58.047630 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.047483    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" containerID="cri-o://7650c165190f3fdb24d25c8c68419b23098326e02677d37f0836cb32b454af74" gracePeriod=30
Oct 31 00:37:58.172008 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.171981    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:37:58.229594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229555    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229670    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229715    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229775    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229811    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229866    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229906    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.229957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.229936    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.230157 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230046    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert\") pod \"de645620-02e8-42ee-8687-02b4892cdc1c\" (UID: \"de645620-02e8-42ee-8687-02b4892cdc1c\") "
Oct 31 00:37:58.230209 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230142    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:37:58.230749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230348    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:37:58.230749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230681    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:37:58.230749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230690    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.230749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230716    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.230749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.230681    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:37:58.232745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.231994    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:37:58.232745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.232705    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "kube-api-access-l9jvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:58.232986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.232961    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:37:58.233360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.233330    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:37:58.233360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.233340    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo" pod "de645620-02e8-42ee-8687-02b4892cdc1c" (UID: "de645620-02e8-42ee-8687-02b4892cdc1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:37:58.331903 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331863    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de645620-02e8-42ee-8687-02b4892cdc1c-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.331903 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331897    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9jvw\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-kube-api-access-l9jvw\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.331903 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331908    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.332158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331917    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.332158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331926    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de645620-02e8-42ee-8687-02b4892cdc1c-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.332158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331936    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de645620-02e8-42ee-8687-02b4892cdc1c-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.332158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.331944    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de645620-02e8-42ee-8687-02b4892cdc1c-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:37:58.481292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.481258    4478 generic.go:356] "Generic (PLEG): container finished" podID="de645620-02e8-42ee-8687-02b4892cdc1c" containerID="8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd" exitCode=0
Oct 31 00:37:58.481505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.481313    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"
Oct 31 00:37:58.481505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.481344    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" event={"ID":"de645620-02e8-42ee-8687-02b4892cdc1c","Type":"ContainerDied","Data":"8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd"}
Oct 31 00:37:58.481505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.481433    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m" event={"ID":"de645620-02e8-42ee-8687-02b4892cdc1c","Type":"ContainerDied","Data":"4b5d8767cc8721a2a0f13d5d5282c5531b9b6230f4c3a43ebc160216684d2c13"}
Oct 31 00:37:58.481505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.481469    4478 scope.go:122] "RemoveContainer" containerID="8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd"
Oct 31 00:37:58.491787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.491772    4478 scope.go:122] "RemoveContainer" containerID="8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd"
Oct 31 00:37:58.492087 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:37:58.492067    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd\": container with ID starting with 8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd not found: ID does not exist" containerID="8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd"
Oct 31 00:37:58.492134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.492097    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd"} err="failed to get container status \"8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd\": rpc error: code = NotFound desc = could not find container \"8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd\": container with ID starting with 8e1a66e240370a3985b9bff09757be993580b754bcc845bb65d222d1199a9bbd not found: ID does not exist"
Oct 31 00:37:58.666550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.666520    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"]
Oct 31 00:37:58.671825 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:58.671801    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-sgb5m"]
Oct 31 00:37:59.118145 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:37:59.118112    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" path="/var/lib/kubelet/pods/de645620-02e8-42ee-8687-02b4892cdc1c/volumes"
Oct 31 00:38:00.497250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.497217    4478 generic.go:356] "Generic (PLEG): container finished" podID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerID="263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef" exitCode=0
Oct 31 00:38:00.497822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.497296    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" event={"ID":"b36bef42-9e41-4da7-a998-103b93a9e39c","Type":"ContainerDied","Data":"263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"}
Oct 31 00:38:00.497914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.497832    4478 scope.go:122] "RemoveContainer" containerID="263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"
Oct 31 00:38:00.499263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.499239    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:00.499338 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.499305    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:00.499374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.499341    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:00.752057 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:00.750159    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:38:01.505258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:01.505222    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" event={"ID":"b36bef42-9e41-4da7-a998-103b93a9e39c","Type":"ContainerStarted","Data":"be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81"}
Oct 31 00:38:01.787360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:01.787269    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:38:01.787360 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:01.787336    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:38:02.170900 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:02.170804    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:38:02.172485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:02.172449    4478 patch_prober.go:29] interesting pod/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:38:02.172638 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:02.172513    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 00:38:03.171631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.171595    4478 patch_prober.go:29] interesting pod/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:38:03.172096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.171648    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 00:38:03.517626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.517590    4478 generic.go:356] "Generic (PLEG): container finished" podID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerID="7650c165190f3fdb24d25c8c68419b23098326e02677d37f0836cb32b454af74" exitCode=0
Oct 31 00:38:03.517810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.517664    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" event={"ID":"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36","Type":"ContainerDied","Data":"7650c165190f3fdb24d25c8c68419b23098326e02677d37f0836cb32b454af74"}
Oct 31 00:38:03.777185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.777118    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:38:03.801046 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801017    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801219 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801074    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801219 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801166    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801219 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801210    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801246    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801304    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801346    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801406    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801633 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801404    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:03.801633 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801461    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:03.801633 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801480    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs\") pod \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\" (UID: \"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36\") "
Oct 31 00:38:03.801788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801685    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:03.801788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801718    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:38:03.801874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.801838    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:03.802280 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.802244    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.802280 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.802274    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.802455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.802290    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.802455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.802305    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.802455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.802320    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.803465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.803445    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "kube-api-access-58bfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:38:03.803575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.803536    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:38:03.804138 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.804114    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:03.804204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.804132    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token" pod "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" (UID: "15e36ff2-88d8-4c0a-8c03-c8271fcf4c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:38:03.903375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.903340    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58bfn\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-kube-api-access-58bfn\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.903375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.903371    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.903593 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.903406    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:03.903593 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:03.903415    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:04.171229 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.171144    4478 patch_prober.go:29] interesting pod/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:38:04.171229 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.171199    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 00:38:04.525047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.525013    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"
Oct 31 00:38:04.525047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.525029    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl" event={"ID":"15e36ff2-88d8-4c0a-8c03-c8271fcf4c36","Type":"ContainerDied","Data":"07c305473e7037eab38561da9ec2b81d7fd22f0e2731062ea5651c420c541dec"}
Oct 31 00:38:04.525573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.525095    4478 scope.go:122] "RemoveContainer" containerID="7650c165190f3fdb24d25c8c68419b23098326e02677d37f0836cb32b454af74"
Oct 31 00:38:04.690771 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.690726    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"]
Oct 31 00:38:04.751999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:04.751960    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-qppxl"]
Oct 31 00:38:05.117707 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:05.117674    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" path="/var/lib/kubelet/pods/15e36ff2-88d8-4c0a-8c03-c8271fcf4c36/volumes"
Oct 31 00:38:05.171159 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:05.171115    4478 patch_prober.go:29] interesting pod/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:38:05.171344 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:05.171188    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 00:38:06.171010 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:06.170978    4478 patch_prober.go:29] interesting pod/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 31 00:38:06.171517 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:06.171039    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 31 00:38:07.176959 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:07.176926    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:38:07.177375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:07.177356    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:38:07.178148 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:07.178129    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:38:14.758464 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:38:14.758421    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:38:16.865212 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.865182    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"]
Oct 31 00:38:16.866882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.866860    4478 cpu_manager.go:414] "containerMap: removing container" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy"
Oct 31 00:38:16.866951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.866883    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy"
Oct 31 00:38:16.866951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.866910    4478 cpu_manager.go:414] "containerMap: removing container" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy"
Oct 31 00:38:16.866951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.866915    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy"
Oct 31 00:38:16.867049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.867003    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="15e36ff2-88d8-4c0a-8c03-c8271fcf4c36" containerName="istio-proxy"
Oct 31 00:38:16.867049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.867016    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="de645620-02e8-42ee-8687-02b4892cdc1c" containerName="istio-proxy"
Oct 31 00:38:16.869270 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.869251    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.923406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.922878    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-726d5b-openshift-default-dockercfg-gw8rv\""
Oct 31 00:38:16.996399 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.996232    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwp5h\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.997967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.996654    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.998218 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.998200    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.998358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.998343    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.998672 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.998655    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.998822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.998801    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.998952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.998937    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.999130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.999116    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:16.999256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:16.999241    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.005209 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.005146    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"]
Oct 31 00:38:17.137181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.137094    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.137416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.137396    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.143807 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.143779    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.145636 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.145610    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.145821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.145806    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146050 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.146032    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwp5h\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.146141    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.146273    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.146395    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.138057    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146801 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.146143    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.146919 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.146885    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.147099 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.139694    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.147313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.147295    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.151610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.151506    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.152236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.152192    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.206889 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.206866    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.207007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.206924    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwp5h\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h\") pod \"e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") " pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.481653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.481604    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:17.913052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.912967    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"]
Oct 31 00:38:17.950927 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:38:17.950901    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd517b74d_a9ea_49f0_9ec2_73358fc5da93.slice/crio-99e0f2b98f01bf2319b57127c7eb917e8391c7064fad38c249c95d5efcde3f90 WatchSource:0}: Error finding container 99e0f2b98f01bf2319b57127c7eb917e8391c7064fad38c249c95d5efcde3f90: Status 404 returned error can't find the container with id 99e0f2b98f01bf2319b57127c7eb917e8391c7064fad38c249c95d5efcde3f90
Oct 31 00:38:17.953268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.953238    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:17.953348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.953308    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:17.953348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:17.953337    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:18.658958 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:18.658918    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" event={"ID":"d517b74d-a9ea-49f0-9ec2-73358fc5da93","Type":"ContainerStarted","Data":"4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"}
Oct 31 00:38:18.659113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:18.658984    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" event={"ID":"d517b74d-a9ea-49f0-9ec2-73358fc5da93","Type":"ContainerStarted","Data":"99e0f2b98f01bf2319b57127c7eb917e8391c7064fad38c249c95d5efcde3f90"}
Oct 31 00:38:18.880775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:18.880728    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podStartSLOduration=2.880712708 podStartE2EDuration="2.880712708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:18.830861772 +0000 UTC m=+2131.847526486" watchObservedRunningTime="2026-10-31 00:38:18.880712708 +0000 UTC m=+2131.897377426"
Oct 31 00:38:19.482734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:19.482705    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:19.488687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:19.488661    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:19.665456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:19.665415    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:19.666463 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:19.666443    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:38:26.689105 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.687144    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"]
Oct 31 00:38:26.707260 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.707206    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:26.747091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.747057    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"]
Oct 31 00:38:26.790406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.790225    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-726d5b-dockercfg-29h7l\""
Oct 31 00:38:26.790406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.790359    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-726d5b\""
Oct 31 00:38:26.814430 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.813949    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g8z6\" (UniqueName: \"kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6\") pod \"payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") " pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:26.815071 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.815045    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") " pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:26.917023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.916990    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g8z6\" (UniqueName: \"kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6\") pod \"payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") " pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:26.917198 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.917031    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") " pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:26.917927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.917900    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume\") pod \"payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") " pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:26.978447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:26.978331    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g8z6\" (UniqueName: \"kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6\") pod \"payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") " pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:27.048926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:27.048893    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:27.432688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:27.432649    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"]
Oct 31 00:38:27.434195 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:38:27.434168    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cd784f_7ce6_430c_b879_d4b7c4b2f89a.slice/crio-a95c80bbae7b414911e4690a74f08254398344fbc18a06fa98ff730592fa0d7e WatchSource:0}: Error finding container a95c80bbae7b414911e4690a74f08254398344fbc18a06fa98ff730592fa0d7e: Status 404 returned error can't find the container with id a95c80bbae7b414911e4690a74f08254398344fbc18a06fa98ff730592fa0d7e
Oct 31 00:38:27.760328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:27.760240    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t" event={"ID":"14cd784f-7ce6-430c-b879-d4b7c4b2f89a","Type":"ContainerStarted","Data":"1a537948f761847c9b5cc051b21e1cb9ecea50d5dc33d697693f89bc669dc996"}
Oct 31 00:38:27.760328 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:27.760297    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t" event={"ID":"14cd784f-7ce6-430c-b879-d4b7c4b2f89a","Type":"ContainerStarted","Data":"a95c80bbae7b414911e4690a74f08254398344fbc18a06fa98ff730592fa0d7e"}
Oct 31 00:38:27.760785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:27.760352    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:27.878861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:27.878792    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t" podStartSLOduration=1.878772898 podStartE2EDuration="1.878772898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:27.876792224 +0000 UTC m=+2140.893456957" watchObservedRunningTime="2026-10-31 00:38:27.878772898 +0000 UTC m=+2140.895437613"
Oct 31 00:38:36.850083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:36.850048    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"]
Oct 31 00:38:38.024186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:38.024155    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"]
Oct 31 00:38:38.024716 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:38.024423    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" containerID="cri-o://a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca" gracePeriod=30
Oct 31 00:38:38.768726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:38.768699    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:38:39.769295 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:39.769262    4478 patch_prober.go:29] interesting pod/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:38:39.769691 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:39.769327    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:38:41.850756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:41.850710    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt" podUID="ae22bad8-5066-4c9e-be89-b41463ac4807" containerName="payload-pre-processing" containerID="cri-o://463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a" gracePeriod=25
Oct 31 00:38:42.111711 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.111639    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:38:42.157899 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.157867    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v\" (UniqueName: \"kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v\") pod \"ae22bad8-5066-4c9e-be89-b41463ac4807\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") "
Oct 31 00:38:42.158093 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.158076    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume\") pod \"ae22bad8-5066-4c9e-be89-b41463ac4807\" (UID: \"ae22bad8-5066-4c9e-be89-b41463ac4807\") "
Oct 31 00:38:42.158497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.158467    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume" pod "ae22bad8-5066-4c9e-be89-b41463ac4807" (UID: "ae22bad8-5066-4c9e-be89-b41463ac4807"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:38:42.158795 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.158624    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae22bad8-5066-4c9e-be89-b41463ac4807-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:42.160295 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.160271    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v" pod "ae22bad8-5066-4c9e-be89-b41463ac4807" (UID: "ae22bad8-5066-4c9e-be89-b41463ac4807"). InnerVolumeSpecName "kube-api-access-crc9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:38:42.260348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.260316    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crc9v\" (UniqueName: \"kubernetes.io/projected/ae22bad8-5066-4c9e-be89-b41463ac4807-kube-api-access-crc9v\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:42.893658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.893622    4478 generic.go:356] "Generic (PLEG): container finished" podID="ae22bad8-5066-4c9e-be89-b41463ac4807" containerID="463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a" exitCode=0
Oct 31 00:38:42.894153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.893676    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt" event={"ID":"ae22bad8-5066-4c9e-be89-b41463ac4807","Type":"ContainerDied","Data":"463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a"}
Oct 31 00:38:42.894153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.893688    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"
Oct 31 00:38:42.894153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.893740    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt" event={"ID":"ae22bad8-5066-4c9e-be89-b41463ac4807","Type":"ContainerDied","Data":"5a7bf261e2a532dec9934ba3764b16b32bff90bcb8b9a523dff97447ba89180a"}
Oct 31 00:38:42.894153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.893764    4478 scope.go:122] "RemoveContainer" containerID="463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a"
Oct 31 00:38:42.903981 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.903963    4478 scope.go:122] "RemoveContainer" containerID="463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a"
Oct 31 00:38:42.904273 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:38:42.904254    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a\": container with ID starting with 463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a not found: ID does not exist" containerID="463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a"
Oct 31 00:38:42.904327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:42.904282    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a"} err="failed to get container status \"463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a\": rpc error: code = NotFound desc = could not find container \"463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a\": container with ID starting with 463241a5d59c8768120360e15d793b06685ddd9b52a1f8a0e8a0728d01f3b53a not found: ID does not exist"
Oct 31 00:38:43.057986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.057954    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"]
Oct 31 00:38:43.107654 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.107628    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-97fc39-f9d9fd5dc-82kxt"]
Oct 31 00:38:43.119243 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.119218    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae22bad8-5066-4c9e-be89-b41463ac4807" path="/var/lib/kubelet/pods/ae22bad8-5066-4c9e-be89-b41463ac4807/volumes"
Oct 31 00:38:43.287573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.287549    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:38:43.371432 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371370    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371483    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371537    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371599    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371660    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371750 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371689    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371774    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:38:43.371834 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371806    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371855    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.371963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.371946    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk\") pod \"9cb1d2f0-7401-47bb-b522-614f0486f30c\" (UID: \"9cb1d2f0-7401-47bb-b522-614f0486f30c\") "
Oct 31 00:38:43.372426 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.372405    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb1d2f0-7401-47bb-b522-614f0486f30c-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.373158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.373093    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:43.373279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.373234    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:43.373393 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.373360    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:43.374091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.374065    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:43.375618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.375580    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "kube-api-access-mq8hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:38:43.376042 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.376016    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:38:43.376133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.376038    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:38:43.376252 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.376234    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo" pod "9cb1d2f0-7401-47bb-b522-614f0486f30c" (UID: "9cb1d2f0-7401-47bb-b522-614f0486f30c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473677    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473703    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473716    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq8hk\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-kube-api-access-mq8hk\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473726    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473734    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473743    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.473753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473751    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.474067 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.473758    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb1d2f0-7401-47bb-b522-614f0486f30c-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:38:43.902519 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.902437    4478 generic.go:356] "Generic (PLEG): container finished" podID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerID="a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca" exitCode=0
Oct 31 00:38:43.902519 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.902492    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"
Oct 31 00:38:43.902982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.902521    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" event={"ID":"9cb1d2f0-7401-47bb-b522-614f0486f30c","Type":"ContainerDied","Data":"a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca"}
Oct 31 00:38:43.902982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.902580    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv" event={"ID":"9cb1d2f0-7401-47bb-b522-614f0486f30c","Type":"ContainerDied","Data":"284e0a4b7c716e369130e3ee78f0d77d4d95cdfdbc986aa45e5ef27197e2be82"}
Oct 31 00:38:43.902982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.902601    4478 scope.go:122] "RemoveContainer" containerID="a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca"
Oct 31 00:38:43.912860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.912844    4478 scope.go:122] "RemoveContainer" containerID="a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca"
Oct 31 00:38:43.913119 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:38:43.913102    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca\": container with ID starting with a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca not found: ID does not exist" containerID="a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca"
Oct 31 00:38:43.913183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:43.913132    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca"} err="failed to get container status \"a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca\": rpc error: code = NotFound desc = could not find container \"a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca\": container with ID starting with a5a450f1ddb9b725a139c1ab3bbd8965eea800dfe967ec5f996a9198d52f64ca not found: ID does not exist"
Oct 31 00:38:44.071171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:44.071139    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"]
Oct 31 00:38:44.120361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:44.120326    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-zsmgv"]
Oct 31 00:38:44.757961 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:38:44.757918    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:38:45.117493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.117465    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" path="/var/lib/kubelet/pods/9cb1d2f0-7401-47bb-b522-614f0486f30c/volumes"
Oct 31 00:38:45.890803 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.890768    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"]
Oct 31 00:38:45.891310 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.891294    4478 cpu_manager.go:414] "containerMap: removing container" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy"
Oct 31 00:38:45.891419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.891311    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy"
Oct 31 00:38:45.891419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.891343    4478 cpu_manager.go:414] "containerMap: removing container" podUID="ae22bad8-5066-4c9e-be89-b41463ac4807" containerName="payload-pre-processing"
Oct 31 00:38:45.891419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.891348    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae22bad8-5066-4c9e-be89-b41463ac4807" containerName="payload-pre-processing"
Oct 31 00:38:45.891582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.891446    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae22bad8-5066-4c9e-be89-b41463ac4807" containerName="payload-pre-processing"
Oct 31 00:38:45.891582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.891458    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb1d2f0-7401-47bb-b522-614f0486f30c" containerName="istio-proxy"
Oct 31 00:38:45.896309 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.896285    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:45.950205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:45.950180    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2c02a9\""
Oct 31 00:38:46.009839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.009808    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2c02a9-dockercfg-dvzqh\""
Oct 31 00:38:46.057417 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.057371    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"]
Oct 31 00:38:46.063964 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.061573    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"]
Oct 31 00:38:46.067457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.067437    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.094587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.094559    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"]
Oct 31 00:38:46.137320 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.137288    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.137703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.137480    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bj54w\" (UniqueName: \"kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.239278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.239237    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bj54w\" (UniqueName: \"kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.239474 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.239310    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.240111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.239490    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.240111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.240030    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhvtq\" (UniqueName: \"kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq\") pod \"payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.240819 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.240790    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.300770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.300741    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bj54w\" (UniqueName: \"kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.341394 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.341350    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.341553 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.341491    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhvtq\" (UniqueName: \"kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq\") pod \"payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.342049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.342031    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.402463 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.402433    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhvtq\" (UniqueName: \"kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq\") pod \"payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.511753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.511675    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:46.695755 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.695710    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:46.894548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.892851    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"]
Oct 31 00:38:46.896344 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:38:46.895425    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod215fbf8a_87ea_4756_b7fc_f5acff649cae.slice/crio-b3f3ec0d93f7cd84479a93d0c947badbc0b31f11f7e6defb04d938fbab79fead WatchSource:0}: Error finding container b3f3ec0d93f7cd84479a93d0c947badbc0b31f11f7e6defb04d938fbab79fead: Status 404 returned error can't find the container with id b3f3ec0d93f7cd84479a93d0c947badbc0b31f11f7e6defb04d938fbab79fead
Oct 31 00:38:46.921538 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:46.921512    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v" event={"ID":"215fbf8a-87ea-4756-b7fc-f5acff649cae","Type":"ContainerStarted","Data":"b3f3ec0d93f7cd84479a93d0c947badbc0b31f11f7e6defb04d938fbab79fead"}
Oct 31 00:38:47.079072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:47.079050    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"]
Oct 31 00:38:47.081355 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:38:47.081329    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd15caa27_bb2c_4cdb_9ded_9d50513a8e72.slice/crio-f0d5cac28f78164a189957e6f54c33db5a8ecae234ba7569f0deec72211ca98c WatchSource:0}: Error finding container f0d5cac28f78164a189957e6f54c33db5a8ecae234ba7569f0deec72211ca98c: Status 404 returned error can't find the container with id f0d5cac28f78164a189957e6f54c33db5a8ecae234ba7569f0deec72211ca98c
Oct 31 00:38:47.929592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:47.929555    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v" event={"ID":"215fbf8a-87ea-4756-b7fc-f5acff649cae","Type":"ContainerStarted","Data":"f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1"}
Oct 31 00:38:47.930063 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:47.929742    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:47.931309 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:47.931282    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6" event={"ID":"d15caa27-bb2c-4cdb-9ded-9d50513a8e72","Type":"ContainerStarted","Data":"5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb"}
Oct 31 00:38:47.931453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:47.931321    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6" event={"ID":"d15caa27-bb2c-4cdb-9ded-9d50513a8e72","Type":"ContainerStarted","Data":"f0d5cac28f78164a189957e6f54c33db5a8ecae234ba7569f0deec72211ca98c"}
Oct 31 00:38:47.931453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:47.931356    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:38:48.054504 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:48.054439    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v" podStartSLOduration=3.05442051 podStartE2EDuration="3.05442051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:48.051223683 +0000 UTC m=+2161.067888395" watchObservedRunningTime="2026-10-31 00:38:48.05442051 +0000 UTC m=+2161.071085223"
Oct 31 00:38:48.182283 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:48.169124    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6" podStartSLOduration=3.169102887 podStartE2EDuration="3.169102887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:48.166041044 +0000 UTC m=+2161.182705822" watchObservedRunningTime="2026-10-31 00:38:48.169102887 +0000 UTC m=+2161.185767602"
Oct 31 00:38:54.621682 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.621645    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"]
Oct 31 00:38:54.627078 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.627061    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686100    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvzd4\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686283 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686139    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686283 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686267    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686289    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686304    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686424 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686333    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686511    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686643 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686601    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.686700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.686653    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.726599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.726571    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"]
Oct 31 00:38:54.780694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.780651    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-1a4f30-openshift-default-dockercfg-ldd42\""
Oct 31 00:38:54.796105 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.796029    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvzd4\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.796690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.796663    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.797210 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.797193    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.797737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.797721    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.798098 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.798084    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.798551 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.798535    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.799423 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.799404    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.799900 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.799883    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.800302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.798042    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.800458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.799754    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.800560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.797671    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.800653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.799276    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.800747 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.800205    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.800874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.800859    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.832404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.806858    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.832404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.825674    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.862413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.857534    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.862413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.857683    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvzd4\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4\") pod \"e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") " pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:54.946008 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:54.945958    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:55.389543 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:55.389519    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"]
Oct 31 00:38:55.391871 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:38:55.391842    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91616be9_2019_4851_a560_5d2979f2f9b1.slice/crio-9a0bf287ff54d0d38a847841132b6dc49445e4061410018135657586b207ce89 WatchSource:0}: Error finding container 9a0bf287ff54d0d38a847841132b6dc49445e4061410018135657586b207ce89: Status 404 returned error can't find the container with id 9a0bf287ff54d0d38a847841132b6dc49445e4061410018135657586b207ce89
Oct 31 00:38:55.394286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:55.394254    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:55.394398 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:55.394319    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:55.394398 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:55.394350    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:38:55.991809 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:55.991770    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" event={"ID":"91616be9-2019-4851-a560-5d2979f2f9b1","Type":"ContainerStarted","Data":"b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"}
Oct 31 00:38:55.992189 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:55.991829    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" event={"ID":"91616be9-2019-4851-a560-5d2979f2f9b1","Type":"ContainerStarted","Data":"9a0bf287ff54d0d38a847841132b6dc49445e4061410018135657586b207ce89"}
Oct 31 00:38:56.149072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.126832    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podStartSLOduration=2.126813823 podStartE2EDuration="2.126813823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:56.125911971 +0000 UTC m=+2169.142576703" watchObservedRunningTime="2026-10-31 00:38:56.126813823 +0000 UTC m=+2169.143478536"
Oct 31 00:38:56.225750 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.225719    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"]
Oct 31 00:38:56.229782 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.229760    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.293101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.293005    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-87b0f8\""
Oct 31 00:38:56.297232 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.297179    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"]
Oct 31 00:38:56.297670 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.297428    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-87b0f8-dockercfg-vvpcl\""
Oct 31 00:38:56.353824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.353788    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z68gh\" (UniqueName: \"kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh\") pod \"payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.353998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.353921    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.455020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.454990    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z68gh\" (UniqueName: \"kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh\") pod \"payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.455204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.455044    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.455795 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.455773    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.515752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.515728    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z68gh\" (UniqueName: \"kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh\") pod \"payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.565450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.565366    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:56.942708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.942670    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"]
Oct 31 00:38:56.944369 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:38:56.944333    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56ce0b99_c09a_475a_beb4_c4ca8f45e130.slice/crio-ab38bbf3194d929dd7a3887c790bb11c63274d5b6a71393798699412222496d1 WatchSource:0}: Error finding container ab38bbf3194d929dd7a3887c790bb11c63274d5b6a71393798699412222496d1: Status 404 returned error can't find the container with id ab38bbf3194d929dd7a3887c790bb11c63274d5b6a71393798699412222496d1
Oct 31 00:38:56.946394 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.946355    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:56.954167 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.954146    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:56.998418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.998395    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" event={"ID":"56ce0b99-c09a-475a-beb4-c4ca8f45e130","Type":"ContainerStarted","Data":"ab38bbf3194d929dd7a3887c790bb11c63274d5b6a71393798699412222496d1"}
Oct 31 00:38:56.998753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.998673    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:56.999678 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:56.999660    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:38:58.006598 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:58.006563    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" event={"ID":"56ce0b99-c09a-475a-beb4-c4ca8f45e130","Type":"ContainerStarted","Data":"662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf"}
Oct 31 00:38:58.007006 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:58.006677    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:38:58.122288 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:58.122240    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" podStartSLOduration=2.122226695 podStartE2EDuration="2.122226695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:38:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:38:58.120675098 +0000 UTC m=+2171.137339811" watchObservedRunningTime="2026-10-31 00:38:58.122226695 +0000 UTC m=+2171.138891407"
Oct 31 00:38:58.941265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:58.941230    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:38:58.941506 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:38:58.941312    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:39:02.707477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.707439    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"]
Oct 31 00:39:02.712591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.712565    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:02.747940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.747909    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdtn\" (UniqueName: \"kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:02.748129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.747967    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:02.774548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.774518    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"]
Oct 31 00:39:02.828640 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.827072    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a4f30\""
Oct 31 00:39:02.860408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.857095    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:02.860408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.857348    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdtn\" (UniqueName: \"kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:02.913521 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.913484    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:02.944410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.942824    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"]
Oct 31 00:39:02.950405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.948712    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:02.979690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.977802    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"]
Oct 31 00:39:02.989066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:02.983718    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdtn\" (UniqueName: \"kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn\") pod \"payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:03.006242 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.003060    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a4f30-dockercfg-xk5w6\""
Oct 31 00:39:03.012475 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.011632    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6g5pf\" (UniqueName: \"kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf\") pod \"payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.012475 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.011715    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.026073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.026049    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:03.113855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.113108    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6g5pf\" (UniqueName: \"kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf\") pod \"payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.113855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.113189    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.114563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.114513    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.175299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.175264    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6g5pf\" (UniqueName: \"kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf\") pod \"payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.295983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.295904    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:03.409007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.408973    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"]
Oct 31 00:39:03.411721 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:39:03.411691    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3836f164_7138_4179_8121_44b930ad8eab.slice/crio-651ddbd0b492d8ad052a3ecafa15e5e3cdd627724628ce9d812d727ac1f005ad WatchSource:0}: Error finding container 651ddbd0b492d8ad052a3ecafa15e5e3cdd627724628ce9d812d727ac1f005ad: Status 404 returned error can't find the container with id 651ddbd0b492d8ad052a3ecafa15e5e3cdd627724628ce9d812d727ac1f005ad
Oct 31 00:39:03.679473 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:03.679429    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"]
Oct 31 00:39:03.680711 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:39:03.680686    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eefc116_ddc5_4430_8833_a87ac24c378e.slice/crio-95ab289cd3afd205cc473343f21a1d795039eae060469687081427ddf7b931d8 WatchSource:0}: Error finding container 95ab289cd3afd205cc473343f21a1d795039eae060469687081427ddf7b931d8: Status 404 returned error can't find the container with id 95ab289cd3afd205cc473343f21a1d795039eae060469687081427ddf7b931d8
Oct 31 00:39:04.045916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.045811    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg" event={"ID":"3836f164-7138-4179-8121-44b930ad8eab","Type":"ContainerStarted","Data":"4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2"}
Oct 31 00:39:04.045916 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.045881    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg" event={"ID":"3836f164-7138-4179-8121-44b930ad8eab","Type":"ContainerStarted","Data":"651ddbd0b492d8ad052a3ecafa15e5e3cdd627724628ce9d812d727ac1f005ad"}
Oct 31 00:39:04.046359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.045955    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:04.048172 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.048147    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29" event={"ID":"7eefc116-ddc5-4430-8833-a87ac24c378e","Type":"ContainerStarted","Data":"c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf"}
Oct 31 00:39:04.048327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.048190    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29" event={"ID":"7eefc116-ddc5-4430-8833-a87ac24c378e","Type":"ContainerStarted","Data":"95ab289cd3afd205cc473343f21a1d795039eae060469687081427ddf7b931d8"}
Oct 31 00:39:04.048327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.048235    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:04.162819 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.162759    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg" podStartSLOduration=2.162742042 podStartE2EDuration="2.162742042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:04.160740605 +0000 UTC m=+2177.177405327" watchObservedRunningTime="2026-10-31 00:39:04.162742042 +0000 UTC m=+2177.179406798"
Oct 31 00:39:04.277494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:04.277281    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29" podStartSLOduration=2.277263968 podStartE2EDuration="2.277263968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:04.276650903 +0000 UTC m=+2177.293315616" watchObservedRunningTime="2026-10-31 00:39:04.277263968 +0000 UTC m=+2177.293928682"
Oct 31 00:39:08.155055 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:08.155023    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 00:39:08.424935 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:08.424912    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 00:39:09.014849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:09.014818    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:39:14.758000 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:39:14.757956    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:39:15.059073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:15.058994    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:39:15.059317 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:15.059295    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:39:40.428335 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.428284    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"]
Oct 31 00:39:40.435776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.435732    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.521405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.502159    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm585\" (UniqueName: \"kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585\") pod \"payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.521405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.502254    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.541408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.541346    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-97fc39-dockercfg-f8nvj\""
Oct 31 00:39:40.541668 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.541535    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-97fc39\""
Oct 31 00:39:40.544200 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.544167    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"]
Oct 31 00:39:40.603435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.603406    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm585\" (UniqueName: \"kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585\") pod \"payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.603572 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.603463    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.604110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.604092    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.664945 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.664921    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm585\" (UniqueName: \"kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585\") pod \"payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:40.779127 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:40.779057    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:41.169108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:41.169078    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"]
Oct 31 00:39:41.171448 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:39:41.171416    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6a605fe_550c_40c0_b1cc_ded0e41a02b1.slice/crio-9d9047cd3a0f03bcc0c37989c32eb71505b98b28ac9916211106ea7d6a946f17 WatchSource:0}: Error finding container 9d9047cd3a0f03bcc0c37989c32eb71505b98b28ac9916211106ea7d6a946f17: Status 404 returned error can't find the container with id 9d9047cd3a0f03bcc0c37989c32eb71505b98b28ac9916211106ea7d6a946f17
Oct 31 00:39:41.310555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:41.310516    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw" event={"ID":"c6a605fe-550c-40c0-b1cc-ded0e41a02b1","Type":"ContainerStarted","Data":"9d9047cd3a0f03bcc0c37989c32eb71505b98b28ac9916211106ea7d6a946f17"}
Oct 31 00:39:44.358236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:44.358143    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw" event={"ID":"c6a605fe-550c-40c0-b1cc-ded0e41a02b1","Type":"ContainerStarted","Data":"8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95"}
Oct 31 00:39:44.360670 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:44.360648    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:44.475924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:44.475863    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw" podStartSLOduration=1.633256892 podStartE2EDuration="4.475811822s" totalImagesPullingTime="2.84255493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:39:44.474963046 +0000 UTC m=+2217.491627758" watchObservedRunningTime="2026-10-31 00:39:44.475811822 +0000 UTC m=+2217.492476534"
Oct 31 00:39:44.791342 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:39:44.791299    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:39:50.112841 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.112808    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": read tcp 10.132.0.2:49450->10.132.0.103:15021: read: connection reset by peer" start-of-body=
Oct 31 00:39:50.113254 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.112869    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": read tcp 10.132.0.2:49450->10.132.0.103:15021: read: connection reset by peer"
Oct 31 00:39:50.418561 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.418468    4478 generic.go:356] "Generic (PLEG): container finished" podID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerID="4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206" exitCode=0
Oct 31 00:39:50.418561 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.418541    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" event={"ID":"d517b74d-a9ea-49f0-9ec2-73358fc5da93","Type":"ContainerDied","Data":"4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"}
Oct 31 00:39:50.418970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.418953    4478 scope.go:122] "RemoveContainer" containerID="4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"
Oct 31 00:39:50.420484 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.420460    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:39:50.420546 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.420530    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:39:50.420583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:50.420566    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:39:51.426900 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:51.426869    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" event={"ID":"d517b74d-a9ea-49f0-9ec2-73358fc5da93","Type":"ContainerStarted","Data":"f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8"}
Oct 31 00:39:51.482820 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:51.482786    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:39:51.483056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:51.483003    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:39:51.483110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:51.483052    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:39:52.482914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:52.482877    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:39:52.483300 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:52.482940    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:39:53.482419 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:53.482368    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:39:53.482600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:53.482441    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:39:54.482303 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:54.482263    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:39:54.482744 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:54.482329    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:39:55.378597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:55.378553    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:39:55.482397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:55.482339    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:39:55.482810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:55.482439    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:39:56.481950 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:56.481914    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Oct 31 00:39:56.482101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:56.481968    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Oct 31 00:39:57.486464 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.486434    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:39:57.486936 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.486910    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:39:57.487681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.487663    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:39:57.493859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.493835    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"]
Oct 31 00:39:57.499699 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.499677    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.553198 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.553175    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-87b0f8-serving-cert\""
Oct 31 00:39:57.583745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.583715    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2npjz\" (UniqueName: \"kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.583898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.583877    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.584140 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.584109    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.584242 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.584171    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.590833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.590811    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"]
Oct 31 00:39:57.685104 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.685060    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.685278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.685112    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.685278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.685155    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2npjz\" (UniqueName: \"kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.685278 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.685257    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.685489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:39:57.685300    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-serving-cert: secret "maas-api-e2e-praxis-a-87b0f8-serving-cert" not found
Oct 31 00:39:57.685489 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:39:57.685372    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls podName:4ccf352b-35e8-4929-aa04-7cff9e08d928 nodeName:}" failed. No retries permitted until 2026-10-31 00:39:58.185350555 +0000 UTC m=+2231.202015262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls") pod "maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" (UID: "4ccf352b-35e8-4929-aa04-7cff9e08d928") : secret "maas-api-e2e-praxis-a-87b0f8-serving-cert" not found
Oct 31 00:39:57.685906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.685883    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.687874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.687852    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:57.754255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:57.754035    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2npjz\" (UniqueName: \"kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:58.190529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:58.190504    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:58.193003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:58.192974    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\") pod \"maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:58.414854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:58.414827    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:39:58.799832 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:58.799799    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"]
Oct 31 00:39:58.801478 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:39:58.801454    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccf352b_35e8_4929_aa04_7cff9e08d928.slice/crio-b9640ba77e13945e4945423b23d2d16910776ca0ea6603f15f7d0ab3515e63a5 WatchSource:0}: Error finding container b9640ba77e13945e4945423b23d2d16910776ca0ea6603f15f7d0ab3515e63a5: Status 404 returned error can't find the container with id b9640ba77e13945e4945423b23d2d16910776ca0ea6603f15f7d0ab3515e63a5
Oct 31 00:39:59.499121 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:39:59.499083    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" event={"ID":"4ccf352b-35e8-4929-aa04-7cff9e08d928","Type":"ContainerStarted","Data":"b9640ba77e13945e4945423b23d2d16910776ca0ea6603f15f7d0ab3515e63a5"}
Oct 31 00:40:00.506700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:00.506657    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" event={"ID":"4ccf352b-35e8-4929-aa04-7cff9e08d928","Type":"ContainerStarted","Data":"37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852"}
Oct 31 00:40:00.507186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:00.506727    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:40:00.634788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:00.634728    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" podStartSLOduration=2.7313278629999997 podStartE2EDuration="3.634713401s" totalImagesPullingTime="903.385538ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:39:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:40:00.632777123 +0000 UTC m=+2233.649441837" watchObservedRunningTime="2026-10-31 00:40:00.634713401 +0000 UTC m=+2233.651378140"
Oct 31 00:40:06.519007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:06.518977    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:40:14.760033 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:40:14.759991    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:40:44.782474 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:40:44.782442    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:40:53.899597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:53.899565    4478 generic.go:356] "Generic (PLEG): container finished" podID="91616be9-2019-4851-a560-5d2979f2f9b1" containerID="b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03" exitCode=0
Oct 31 00:40:53.900047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:53.899629    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" event={"ID":"91616be9-2019-4851-a560-5d2979f2f9b1","Type":"ContainerDied","Data":"b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"}
Oct 31 00:40:53.900047 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:53.900008    4478 scope.go:122] "RemoveContainer" containerID="b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"
Oct 31 00:40:53.901434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:53.901407    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:40:53.901512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:53.901473    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:40:53.901512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:53.901505    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:40:54.907514 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:54.907473    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" event={"ID":"91616be9-2019-4851-a560-5d2979f2f9b1","Type":"ContainerStarted","Data":"a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f"}
Oct 31 00:40:54.959789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:54.946779    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:40:54.962655 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:54.962620    4478 patch_prober.go:29] interesting pod/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 00:40:54.962818 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:54.962686    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 00:40:55.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:55.954497    4478 patch_prober.go:29] interesting pod/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 00:40:55.955146 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:55.955116    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 00:40:56.953405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:56.952781    4478 patch_prober.go:29] interesting pod/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 00:40:56.953405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:56.952861    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 00:40:56.998817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:56.998783    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:40:57.954434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:57.952762    4478 patch_prober.go:29] interesting pod/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 00:40:57.954434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:57.952828    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 00:40:58.600730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:58.600700    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"]
Oct 31 00:40:58.975977 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:58.975928    4478 patch_prober.go:29] interesting pod/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 00:40:58.976256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:58.976216    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 00:40:59.949793 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:59.949755    4478 patch_prober.go:29] interesting pod/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Oct 31 00:40:59.950767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:40:59.950355    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Oct 31 00:41:00.951742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:00.951712    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:41:00.952920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:00.952902    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:41:03.601786 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:03.601724    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s" podUID="624d5193-dbd6-4d88-94f9-3154f5f2c7cb" containerName="payload-processing" containerID="cri-o://eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6" gracePeriod=25
Oct 31 00:41:03.875236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:03.875212    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:41:04.011687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.011646    4478 generic.go:356] "Generic (PLEG): container finished" podID="624d5193-dbd6-4d88-94f9-3154f5f2c7cb" containerID="eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6" exitCode=0
Oct 31 00:41:04.011857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.011711    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"
Oct 31 00:41:04.011857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.011733    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s" event={"ID":"624d5193-dbd6-4d88-94f9-3154f5f2c7cb","Type":"ContainerDied","Data":"eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6"}
Oct 31 00:41:04.011857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.011807    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s" event={"ID":"624d5193-dbd6-4d88-94f9-3154f5f2c7cb","Type":"ContainerDied","Data":"c1b3ea69b264439d151d5a801c8cf50cbb792d28c185878e49171c702913f3ac"}
Oct 31 00:41:04.011857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.011838    4478 scope.go:122] "RemoveContainer" containerID="eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6"
Oct 31 00:41:04.016355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.016328    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume\") pod \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") "
Oct 31 00:41:04.016627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.016608    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4\" (UniqueName: \"kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4\") pod \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\" (UID: \"624d5193-dbd6-4d88-94f9-3154f5f2c7cb\") "
Oct 31 00:41:04.016804 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.016766    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume" pod "624d5193-dbd6-4d88-94f9-3154f5f2c7cb" (UID: "624d5193-dbd6-4d88-94f9-3154f5f2c7cb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:04.017163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.017138    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:04.018852 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.018828    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4" pod "624d5193-dbd6-4d88-94f9-3154f5f2c7cb" (UID: "624d5193-dbd6-4d88-94f9-3154f5f2c7cb"). InnerVolumeSpecName "kube-api-access-8tmv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:04.023661 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.023642    4478 scope.go:122] "RemoveContainer" containerID="eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6"
Oct 31 00:41:04.024070 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:04.024047    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6\": container with ID starting with eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6 not found: ID does not exist" containerID="eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6"
Oct 31 00:41:04.024169 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.024083    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6"} err="failed to get container status \"eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6\": rpc error: code = NotFound desc = could not find container \"eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6\": container with ID starting with eae1e3465861d05baf8cbb1cd542d975c9fa7ed4a80e3e03adc82fd8b0364ce6 not found: ID does not exist"
Oct 31 00:41:04.118151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.118089    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tmv4\" (UniqueName: \"kubernetes.io/projected/624d5193-dbd6-4d88-94f9-3154f5f2c7cb-kube-api-access-8tmv4\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:04.433516 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.433482    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"]
Oct 31 00:41:04.482778 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:04.482743    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-752be8-7644cdf87c-8xh6s"]
Oct 31 00:41:05.118161 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:05.118127    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="624d5193-dbd6-4d88-94f9-3154f5f2c7cb" path="/var/lib/kubelet/pods/624d5193-dbd6-4d88-94f9-3154f5f2c7cb/volumes"
Oct 31 00:41:07.241414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.211565    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"]
Oct 31 00:41:07.241414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.211908    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" podUID="ec54d20e-c588-4c99-8a58-1157e796616a" containerName="maas-api" containerID="cri-o://91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23" gracePeriod=30
Oct 31 00:41:07.612098 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.612071    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:41:07.738633 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.738606    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs\") pod \"ec54d20e-c588-4c99-8a58-1157e796616a\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") "
Oct 31 00:41:07.738809 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.738763    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n\" (UniqueName: \"kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n\") pod \"ec54d20e-c588-4c99-8a58-1157e796616a\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") "
Oct 31 00:41:07.738809 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.738794    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca\") pod \"ec54d20e-c588-4c99-8a58-1157e796616a\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") "
Oct 31 00:41:07.738954 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.738867    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\") pod \"ec54d20e-c588-4c99-8a58-1157e796616a\" (UID: \"ec54d20e-c588-4c99-8a58-1157e796616a\") "
Oct 31 00:41:07.739284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.739251    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca" pod "ec54d20e-c588-4c99-8a58-1157e796616a" (UID: "ec54d20e-c588-4c99-8a58-1157e796616a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:07.741238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.741208    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs" pod "ec54d20e-c588-4c99-8a58-1157e796616a" (UID: "ec54d20e-c588-4c99-8a58-1157e796616a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:41:07.741350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.741283    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls" pod "ec54d20e-c588-4c99-8a58-1157e796616a" (UID: "ec54d20e-c588-4c99-8a58-1157e796616a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:41:07.742924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.742894    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n" pod "ec54d20e-c588-4c99-8a58-1157e796616a" (UID: "ec54d20e-c588-4c99-8a58-1157e796616a"). InnerVolumeSpecName "kube-api-access-lwv5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:07.840327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.840252    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:07.840327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.840282    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lwv5n\" (UniqueName: \"kubernetes.io/projected/ec54d20e-c588-4c99-8a58-1157e796616a-kube-api-access-lwv5n\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:07.840327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.840295    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec54d20e-c588-4c99-8a58-1157e796616a-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:07.840327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:07.840303    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec54d20e-c588-4c99-8a58-1157e796616a-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:08.041374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.041327    4478 generic.go:356] "Generic (PLEG): container finished" podID="ec54d20e-c588-4c99-8a58-1157e796616a" containerID="91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23" exitCode=0
Oct 31 00:41:08.041578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.041427    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" event={"ID":"ec54d20e-c588-4c99-8a58-1157e796616a","Type":"ContainerDied","Data":"91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23"}
Oct 31 00:41:08.041578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.041446    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"
Oct 31 00:41:08.041578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.041495    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787" event={"ID":"ec54d20e-c588-4c99-8a58-1157e796616a","Type":"ContainerDied","Data":"0073eaa758678c66d7cb61ed012a2ad4ec7514a782a10c7535ab623956309882"}
Oct 31 00:41:08.041578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.041520    4478 scope.go:122] "RemoveContainer" containerID="91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23"
Oct 31 00:41:08.057749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.057725    4478 scope.go:122] "RemoveContainer" containerID="91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23"
Oct 31 00:41:08.060631 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:08.060595    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23\": container with ID starting with 91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23 not found: ID does not exist" containerID="91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23"
Oct 31 00:41:08.060740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.060644    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23"} err="failed to get container status \"91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23\": rpc error: code = NotFound desc = could not find container \"91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23\": container with ID starting with 91315714fcb32eadcfe55d001c745946890b797b3036be7924a18f1d85afee23 not found: ID does not exist"
Oct 31 00:41:08.320250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.320203    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"]
Oct 31 00:41:08.323185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:08.323140    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bba5ed-dd95d898d-tj787"]
Oct 31 00:41:09.117541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:09.117511    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec54d20e-c588-4c99-8a58-1157e796616a" path="/var/lib/kubelet/pods/ec54d20e-c588-4c99-8a58-1157e796616a/volumes"
Oct 31 00:41:14.621533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:14.621495    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"]
Oct 31 00:41:14.672887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:14.672851    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"]
Oct 31 00:41:14.763154 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:14.763115    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:41:15.210548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:15.210519    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"]
Oct 31 00:41:19.622165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:19.622120    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6" podUID="d15caa27-bb2c-4cdb-9ded-9d50513a8e72" containerName="payload-processing" containerID="cri-o://5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb" gracePeriod=25
Oct 31 00:41:19.673490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:19.673446    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v" podUID="215fbf8a-87ea-4756-b7fc-f5acff649cae" containerName="payload-pre-processing" containerID="cri-o://f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1" gracePeriod=25
Oct 31 00:41:20.019113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.019081    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:41:20.061451 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.061425    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:41:20.152888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.152792    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq\" (UniqueName: \"kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq\") pod \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") "
Oct 31 00:41:20.153740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.153059    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume\") pod \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\" (UID: \"d15caa27-bb2c-4cdb-9ded-9d50513a8e72\") "
Oct 31 00:41:20.153974 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.153947    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume" pod "d15caa27-bb2c-4cdb-9ded-9d50513a8e72" (UID: "d15caa27-bb2c-4cdb-9ded-9d50513a8e72"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:20.155798 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.155766    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq" pod "d15caa27-bb2c-4cdb-9ded-9d50513a8e72" (UID: "d15caa27-bb2c-4cdb-9ded-9d50513a8e72"). InnerVolumeSpecName "kube-api-access-bhvtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:20.173212 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.173166    4478 generic.go:356] "Generic (PLEG): container finished" podID="215fbf8a-87ea-4756-b7fc-f5acff649cae" containerID="f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1" exitCode=0
Oct 31 00:41:20.173362 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.173227    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"
Oct 31 00:41:20.173362 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.173357    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v" event={"ID":"215fbf8a-87ea-4756-b7fc-f5acff649cae","Type":"ContainerDied","Data":"f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1"}
Oct 31 00:41:20.173614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.173447    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v" event={"ID":"215fbf8a-87ea-4756-b7fc-f5acff649cae","Type":"ContainerDied","Data":"b3f3ec0d93f7cd84479a93d0c947badbc0b31f11f7e6defb04d938fbab79fead"}
Oct 31 00:41:20.173614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.173475    4478 scope.go:122] "RemoveContainer" containerID="f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1"
Oct 31 00:41:20.175314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.175274    4478 generic.go:356] "Generic (PLEG): container finished" podID="d15caa27-bb2c-4cdb-9ded-9d50513a8e72" containerID="5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb" exitCode=0
Oct 31 00:41:20.175456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.175347    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6" event={"ID":"d15caa27-bb2c-4cdb-9ded-9d50513a8e72","Type":"ContainerDied","Data":"5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb"}
Oct 31 00:41:20.175456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.175412    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6" event={"ID":"d15caa27-bb2c-4cdb-9ded-9d50513a8e72","Type":"ContainerDied","Data":"f0d5cac28f78164a189957e6f54c33db5a8ecae234ba7569f0deec72211ca98c"}
Oct 31 00:41:20.175589 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.175507    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"
Oct 31 00:41:20.186356 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.186337    4478 scope.go:122] "RemoveContainer" containerID="f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1"
Oct 31 00:41:20.186893 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:20.186868    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1\": container with ID starting with f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1 not found: ID does not exist" containerID="f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1"
Oct 31 00:41:20.187001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.186897    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1"} err="failed to get container status \"f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1\": rpc error: code = NotFound desc = could not find container \"f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1\": container with ID starting with f595a94215cb559abf4b544f021a46d11527a9ed96282071a46f4512b82799f1 not found: ID does not exist"
Oct 31 00:41:20.187001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.186916    4478 scope.go:122] "RemoveContainer" containerID="5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb"
Oct 31 00:41:20.195524 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.195509    4478 scope.go:122] "RemoveContainer" containerID="5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb"
Oct 31 00:41:20.195803 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:20.195784    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb\": container with ID starting with 5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb not found: ID does not exist" containerID="5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb"
Oct 31 00:41:20.195872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.195808    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb"} err="failed to get container status \"5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb\": rpc error: code = NotFound desc = could not find container \"5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb\": container with ID starting with 5b741de47155eaf791142faef0ecf39f939913f3000e70d09babdc0d54412cbb not found: ID does not exist"
Oct 31 00:41:20.254014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.253994    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w\" (UniqueName: \"kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w\") pod \"215fbf8a-87ea-4756-b7fc-f5acff649cae\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") "
Oct 31 00:41:20.254124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.254097    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume\") pod \"215fbf8a-87ea-4756-b7fc-f5acff649cae\" (UID: \"215fbf8a-87ea-4756-b7fc-f5acff649cae\") "
Oct 31 00:41:20.254788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.254472    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume" pod "215fbf8a-87ea-4756-b7fc-f5acff649cae" (UID: "215fbf8a-87ea-4756-b7fc-f5acff649cae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:20.254788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.254719    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:20.254917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.254872    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/215fbf8a-87ea-4756-b7fc-f5acff649cae-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:20.254917 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.254887    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhvtq\" (UniqueName: \"kubernetes.io/projected/d15caa27-bb2c-4cdb-9ded-9d50513a8e72-kube-api-access-bhvtq\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:20.256873 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.256848    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w" pod "215fbf8a-87ea-4756-b7fc-f5acff649cae" (UID: "215fbf8a-87ea-4756-b7fc-f5acff649cae"). InnerVolumeSpecName "kube-api-access-bj54w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:20.356286 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.356254    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bj54w\" (UniqueName: \"kubernetes.io/projected/215fbf8a-87ea-4756-b7fc-f5acff649cae-kube-api-access-bj54w\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:20.423039 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.422968    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"]
Oct 31 00:41:20.425403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.425366    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-5db8c685c-sprt6"]
Oct 31 00:41:20.597641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.597606    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"]
Oct 31 00:41:20.601908 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:20.601886    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-59b8b5c7cf-blh5v"]
Oct 31 00:41:21.124125 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:21.124083    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="215fbf8a-87ea-4756-b7fc-f5acff649cae" path="/var/lib/kubelet/pods/215fbf8a-87ea-4756-b7fc-f5acff649cae/volumes"
Oct 31 00:41:21.125344 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:21.124764    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d15caa27-bb2c-4cdb-9ded-9d50513a8e72" path="/var/lib/kubelet/pods/d15caa27-bb2c-4cdb-9ded-9d50513a8e72/volumes"
Oct 31 00:41:30.231267 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.231229    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2" podUID="c0793873-7f1d-42b3-8343-efd7e744bad2" containerName="main" containerID="cri-o://e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806" gracePeriod=45
Oct 31 00:41:30.616473 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.616447    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:41:30.700076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700043    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv\" (UniqueName: \"kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv\") pod \"c0793873-7f1d-42b3-8343-efd7e744bad2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") "
Oct 31 00:41:30.700266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700097    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home\") pod \"c0793873-7f1d-42b3-8343-efd7e744bad2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") "
Oct 31 00:41:30.700266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700148    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache\") pod \"c0793873-7f1d-42b3-8343-efd7e744bad2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") "
Oct 31 00:41:30.700266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700209    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs\") pod \"c0793873-7f1d-42b3-8343-efd7e744bad2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") "
Oct 31 00:41:30.700266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700260    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm\") pod \"c0793873-7f1d-42b3-8343-efd7e744bad2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") "
Oct 31 00:41:30.700530 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700282    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir\") pod \"c0793873-7f1d-42b3-8343-efd7e744bad2\" (UID: \"c0793873-7f1d-42b3-8343-efd7e744bad2\") "
Oct 31 00:41:30.700530 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700375    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home" pod "c0793873-7f1d-42b3-8343-efd7e744bad2" (UID: "c0793873-7f1d-42b3-8343-efd7e744bad2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:41:30.700530 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700493    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache" pod "c0793873-7f1d-42b3-8343-efd7e744bad2" (UID: "c0793873-7f1d-42b3-8343-efd7e744bad2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:41:30.700663 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700575    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir" pod "c0793873-7f1d-42b3-8343-efd7e744bad2" (UID: "c0793873-7f1d-42b3-8343-efd7e744bad2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:41:30.700773 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700755    4478 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-model-cache\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:30.700846 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700779    4478 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-tmp-dir\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:30.700846 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.700792    4478 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-home\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:30.702574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.702545    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm" pod "c0793873-7f1d-42b3-8343-efd7e744bad2" (UID: "c0793873-7f1d-42b3-8343-efd7e744bad2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:41:30.702702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.702660    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs" pod "c0793873-7f1d-42b3-8343-efd7e744bad2" (UID: "c0793873-7f1d-42b3-8343-efd7e744bad2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:41:30.702702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.702681    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv" pod "c0793873-7f1d-42b3-8343-efd7e744bad2" (UID: "c0793873-7f1d-42b3-8343-efd7e744bad2"). InnerVolumeSpecName "kube-api-access-rw6nv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:30.802147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.802118    4478 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0793873-7f1d-42b3-8343-efd7e744bad2-dshm\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:30.802147 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.802144    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rw6nv\" (UniqueName: \"kubernetes.io/projected/c0793873-7f1d-42b3-8343-efd7e744bad2-kube-api-access-rw6nv\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:30.802344 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.802156    4478 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0793873-7f1d-42b3-8343-efd7e744bad2-tls-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:30.820221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820180    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"]
Oct 31 00:41:30.820824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820804    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d15caa27-bb2c-4cdb-9ded-9d50513a8e72" containerName="payload-processing"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820825    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d15caa27-bb2c-4cdb-9ded-9d50513a8e72" containerName="payload-processing"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820860    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c0793873-7f1d-42b3-8343-efd7e744bad2" containerName="main"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820867    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0793873-7f1d-42b3-8343-efd7e744bad2" containerName="main"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820908    4478 cpu_manager.go:414] "containerMap: removing container" podUID="215fbf8a-87ea-4756-b7fc-f5acff649cae" containerName="payload-pre-processing"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820916    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="215fbf8a-87ea-4756-b7fc-f5acff649cae" containerName="payload-pre-processing"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820926    4478 cpu_manager.go:414] "containerMap: removing container" podUID="ec54d20e-c588-4c99-8a58-1157e796616a" containerName="maas-api"
Oct 31 00:41:30.820940 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820933    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec54d20e-c588-4c99-8a58-1157e796616a" containerName="maas-api"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820944    4478 cpu_manager.go:414] "containerMap: removing container" podUID="624d5193-dbd6-4d88-94f9-3154f5f2c7cb" containerName="payload-processing"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.820952    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="624d5193-dbd6-4d88-94f9-3154f5f2c7cb" containerName="payload-processing"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.821105    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d15caa27-bb2c-4cdb-9ded-9d50513a8e72" containerName="payload-processing"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.821124    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0793873-7f1d-42b3-8343-efd7e744bad2" containerName="main"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.821138    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="215fbf8a-87ea-4756-b7fc-f5acff649cae" containerName="payload-pre-processing"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.821148    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec54d20e-c588-4c99-8a58-1157e796616a" containerName="maas-api"
Oct 31 00:41:30.821271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.821157    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="624d5193-dbd6-4d88-94f9-3154f5f2c7cb" containerName="payload-processing"
Oct 31 00:41:30.824728 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.824704    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:30.886011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.880748    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-868b06-dockercfg-dx8sr\""
Oct 31 00:41:30.886011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.885909    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"]
Oct 31 00:41:30.893445 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.886606    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-868b06\""
Oct 31 00:41:30.909416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.903848    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9xdh\" (UniqueName: \"kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh\") pod \"payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:30.909416 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:30.903981    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:31.005571 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.005538    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:31.005740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.005651    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9xdh\" (UniqueName: \"kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh\") pod \"payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:31.006236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.006216    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:31.067289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.067255    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9xdh\" (UniqueName: \"kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh\") pod \"payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:31.139174 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.139101    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:31.268922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.268889    4478 generic.go:356] "Generic (PLEG): container finished" podID="c0793873-7f1d-42b3-8343-efd7e744bad2" containerID="e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806" exitCode=0
Oct 31 00:41:31.269415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.268939    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"
Oct 31 00:41:31.269415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.269073    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2" event={"ID":"c0793873-7f1d-42b3-8343-efd7e744bad2","Type":"ContainerDied","Data":"e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806"}
Oct 31 00:41:31.269415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.269107    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2" event={"ID":"c0793873-7f1d-42b3-8343-efd7e744bad2","Type":"ContainerDied","Data":"b9cb32d0c6e1aed91ed0383c9c73d4d9c64f28cc3980d76829b3973e5f713c7e"}
Oct 31 00:41:31.269415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.269127    4478 scope.go:122] "RemoveContainer" containerID="e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806"
Oct 31 00:41:31.280316 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.280275    4478 scope.go:122] "RemoveContainer" containerID="e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806"
Oct 31 00:41:31.280707 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:31.280678    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806\": container with ID starting with e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806 not found: ID does not exist" containerID="e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806"
Oct 31 00:41:31.280784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.280721    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806"} err="failed to get container status \"e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806\": rpc error: code = NotFound desc = could not find container \"e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806\": container with ID starting with e783c4f28565e382d77281827b39e81aed0f76a386f772ebaf99e3ebdab85806 not found: ID does not exist"
Oct 31 00:41:31.441734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.441697    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"]
Oct 31 00:41:31.445518 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.445491    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-bba5ed/ipp-route-bba5ed-kserve-775c559cd5-62rz2"]
Oct 31 00:41:31.556011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:31.555988    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"]
Oct 31 00:41:31.559139 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:41:31.559098    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfa89316_903b_4485_bb66_6e882970bab7.slice/crio-4871b3b94ff2380c17a5f04543ca9ae7e7d138235fd5df5c4e8dd438ff40853b WatchSource:0}: Error finding container 4871b3b94ff2380c17a5f04543ca9ae7e7d138235fd5df5c4e8dd438ff40853b: Status 404 returned error can't find the container with id 4871b3b94ff2380c17a5f04543ca9ae7e7d138235fd5df5c4e8dd438ff40853b
Oct 31 00:41:32.284271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:32.284224    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9" event={"ID":"bfa89316-903b-4485-bb66-6e882970bab7","Type":"ContainerStarted","Data":"dca328233ec6a3299d8c8f2f4d42c10bbf3b9e7344baf81c5fbf3aae4f91c385"}
Oct 31 00:41:32.284756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:32.284328    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:32.284756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:32.284352    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9" event={"ID":"bfa89316-903b-4485-bb66-6e882970bab7","Type":"ContainerStarted","Data":"4871b3b94ff2380c17a5f04543ca9ae7e7d138235fd5df5c4e8dd438ff40853b"}
Oct 31 00:41:32.406505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:32.406453    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9" podStartSLOduration=2.406439645 podStartE2EDuration="2.406439645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:41:32.403468093 +0000 UTC m=+2325.420132818" watchObservedRunningTime="2026-10-31 00:41:32.406439645 +0000 UTC m=+2325.423104358"
Oct 31 00:41:33.119019 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:33.118988    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0793873-7f1d-42b3-8343-efd7e744bad2" path="/var/lib/kubelet/pods/c0793873-7f1d-42b3-8343-efd7e744bad2/volumes"
Oct 31 00:41:43.295261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:43.295219    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:41:43.316478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:43.316450    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"]
Oct 31 00:41:43.413790 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:43.413751    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"]
Oct 31 00:41:44.760187 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:44.760142    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:41:48.316798 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.316748    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29" podUID="7eefc116-ddc5-4430-8833-a87ac24c378e" containerName="payload-processing" containerID="cri-o://c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf" gracePeriod=25
Oct 31 00:41:48.414107 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.414043    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg" podUID="3836f164-7138-4179-8121-44b930ad8eab" containerName="payload-pre-processing" containerID="cri-o://4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2" gracePeriod=25
Oct 31 00:41:48.694739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.694706    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:41:48.716957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.716935    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:41:48.784063 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.784024    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf\" (UniqueName: \"kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf\") pod \"7eefc116-ddc5-4430-8833-a87ac24c378e\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") "
Oct 31 00:41:48.784241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.784104    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn\" (UniqueName: \"kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn\") pod \"3836f164-7138-4179-8121-44b930ad8eab\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") "
Oct 31 00:41:48.784241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.784202    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume\") pod \"3836f164-7138-4179-8121-44b930ad8eab\" (UID: \"3836f164-7138-4179-8121-44b930ad8eab\") "
Oct 31 00:41:48.784483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.784238    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume\") pod \"7eefc116-ddc5-4430-8833-a87ac24c378e\" (UID: \"7eefc116-ddc5-4430-8833-a87ac24c378e\") "
Oct 31 00:41:48.785009 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.784977    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume" pod "3836f164-7138-4179-8121-44b930ad8eab" (UID: "3836f164-7138-4179-8121-44b930ad8eab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:48.785654 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.785245    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume" pod "7eefc116-ddc5-4430-8833-a87ac24c378e" (UID: "7eefc116-ddc5-4430-8833-a87ac24c378e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:41:48.787850 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.787822    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf" pod "7eefc116-ddc5-4430-8833-a87ac24c378e" (UID: "7eefc116-ddc5-4430-8833-a87ac24c378e"). InnerVolumeSpecName "kube-api-access-6g5pf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:48.790068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.790029    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn" pod "3836f164-7138-4179-8121-44b930ad8eab" (UID: "3836f164-7138-4179-8121-44b930ad8eab"). InnerVolumeSpecName "kube-api-access-6jdtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:41:48.885897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.885819    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6g5pf\" (UniqueName: \"kubernetes.io/projected/7eefc116-ddc5-4430-8833-a87ac24c378e-kube-api-access-6g5pf\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:48.885897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.885848    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jdtn\" (UniqueName: \"kubernetes.io/projected/3836f164-7138-4179-8121-44b930ad8eab-kube-api-access-6jdtn\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:48.885897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.885864    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3836f164-7138-4179-8121-44b930ad8eab-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:48.885897 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:48.885873    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7eefc116-ddc5-4430-8833-a87ac24c378e-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:41:49.387337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.387301    4478 generic.go:356] "Generic (PLEG): container finished" podID="7eefc116-ddc5-4430-8833-a87ac24c378e" containerID="c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf" exitCode=0
Oct 31 00:41:49.387785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.387358    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"
Oct 31 00:41:49.387785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.387390    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29" event={"ID":"7eefc116-ddc5-4430-8833-a87ac24c378e","Type":"ContainerDied","Data":"c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf"}
Oct 31 00:41:49.387785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.387448    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29" event={"ID":"7eefc116-ddc5-4430-8833-a87ac24c378e","Type":"ContainerDied","Data":"95ab289cd3afd205cc473343f21a1d795039eae060469687081427ddf7b931d8"}
Oct 31 00:41:49.387785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.387470    4478 scope.go:122] "RemoveContainer" containerID="c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf"
Oct 31 00:41:49.389049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.389023    4478 generic.go:356] "Generic (PLEG): container finished" podID="3836f164-7138-4179-8121-44b930ad8eab" containerID="4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2" exitCode=0
Oct 31 00:41:49.389170 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.389084    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg" event={"ID":"3836f164-7138-4179-8121-44b930ad8eab","Type":"ContainerDied","Data":"4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2"}
Oct 31 00:41:49.389170 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.389087    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"
Oct 31 00:41:49.389170 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.389117    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg" event={"ID":"3836f164-7138-4179-8121-44b930ad8eab","Type":"ContainerDied","Data":"651ddbd0b492d8ad052a3ecafa15e5e3cdd627724628ce9d812d727ac1f005ad"}
Oct 31 00:41:49.396938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.396914    4478 scope.go:122] "RemoveContainer" containerID="c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf"
Oct 31 00:41:49.397190 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:49.397162    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf\": container with ID starting with c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf not found: ID does not exist" containerID="c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf"
Oct 31 00:41:49.397271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.397200    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf"} err="failed to get container status \"c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf\": rpc error: code = NotFound desc = could not find container \"c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf\": container with ID starting with c7be7ae0b88a3f47f7eb9b9434c8710bf126a3f3e754d20ed3673b465b8290bf not found: ID does not exist"
Oct 31 00:41:49.397271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.397222    4478 scope.go:122] "RemoveContainer" containerID="4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2"
Oct 31 00:41:49.405420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.405405    4478 scope.go:122] "RemoveContainer" containerID="4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2"
Oct 31 00:41:49.405680 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:41:49.405663    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2\": container with ID starting with 4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2 not found: ID does not exist" containerID="4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2"
Oct 31 00:41:49.405724 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.405688    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2"} err="failed to get container status \"4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2\": rpc error: code = NotFound desc = could not find container \"4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2\": container with ID starting with 4c52f0a732f15babebecda352d94956961b2772856dab25ce05ca87a493785f2 not found: ID does not exist"
Oct 31 00:41:49.555577 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.555547    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"]
Oct 31 00:41:49.603372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.603302    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-74d66f7d5-j9p29"]
Oct 31 00:41:49.668124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.668093    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"]
Oct 31 00:41:49.670999 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:49.670979    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1a4f30-645bd9c455-ppgfg"]
Oct 31 00:41:51.117073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:51.117040    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3836f164-7138-4179-8121-44b930ad8eab" path="/var/lib/kubelet/pods/3836f164-7138-4179-8121-44b930ad8eab/volumes"
Oct 31 00:41:51.117462 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:41:51.117426    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7eefc116-ddc5-4430-8833-a87ac24c378e" path="/var/lib/kubelet/pods/7eefc116-ddc5-4430-8833-a87ac24c378e/volumes"
Oct 31 00:42:01.887942 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:01.887909    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"]
Oct 31 00:42:01.888541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:01.888131    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" podUID="30ab8f75-f762-4a19-aedc-51919c61766a" containerName="maas-api" containerID="cri-o://77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9" gracePeriod=30
Oct 31 00:42:02.203128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.203101    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:42:02.265064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.265033    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t\" (UniqueName: \"kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t\") pod \"30ab8f75-f762-4a19-aedc-51919c61766a\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") "
Oct 31 00:42:02.265221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.265072    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca\") pod \"30ab8f75-f762-4a19-aedc-51919c61766a\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") "
Oct 31 00:42:02.265291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.265265    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs\") pod \"30ab8f75-f762-4a19-aedc-51919c61766a\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") "
Oct 31 00:42:02.265331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.265320    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\") pod \"30ab8f75-f762-4a19-aedc-51919c61766a\" (UID: \"30ab8f75-f762-4a19-aedc-51919c61766a\") "
Oct 31 00:42:02.265458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.265430    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca" pod "30ab8f75-f762-4a19-aedc-51919c61766a" (UID: "30ab8f75-f762-4a19-aedc-51919c61766a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:02.266295 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.266008    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/30ab8f75-f762-4a19-aedc-51919c61766a-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:02.267929 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.267896    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls" pod "30ab8f75-f762-4a19-aedc-51919c61766a" (UID: "30ab8f75-f762-4a19-aedc-51919c61766a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:42:02.268645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.268617    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t" pod "30ab8f75-f762-4a19-aedc-51919c61766a" (UID: "30ab8f75-f762-4a19-aedc-51919c61766a"). InnerVolumeSpecName "kube-api-access-xxk4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:02.313411 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.297548    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs" pod "30ab8f75-f762-4a19-aedc-51919c61766a" (UID: "30ab8f75-f762-4a19-aedc-51919c61766a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:42:02.358721 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.358688    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"]
Oct 31 00:42:02.358921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.358900    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw" podUID="c6a605fe-550c-40c0-b1cc-ded0e41a02b1" containerName="payload-processing" containerID="cri-o://8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95" gracePeriod=30
Oct 31 00:42:02.368167 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.368144    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxk4t\" (UniqueName: \"kubernetes.io/projected/30ab8f75-f762-4a19-aedc-51919c61766a-kube-api-access-xxk4t\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:02.368272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.368193    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:02.368272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.368210    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/30ab8f75-f762-4a19-aedc-51919c61766a-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:02.512641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.497080    4478 generic.go:356] "Generic (PLEG): container finished" podID="30ab8f75-f762-4a19-aedc-51919c61766a" containerID="77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9" exitCode=0
Oct 31 00:42:02.512641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.497156    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" event={"ID":"30ab8f75-f762-4a19-aedc-51919c61766a","Type":"ContainerDied","Data":"77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9"}
Oct 31 00:42:02.512641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.497208    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c" event={"ID":"30ab8f75-f762-4a19-aedc-51919c61766a","Type":"ContainerDied","Data":"099ce724e789cdcc833485bcd949f1f86cd981696cfe294bc4f330aca36308d7"}
Oct 31 00:42:02.512641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.497237    4478 scope.go:122] "RemoveContainer" containerID="77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9"
Oct 31 00:42:02.512641 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.497429    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"
Oct 31 00:42:02.518237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.515260    4478 scope.go:122] "RemoveContainer" containerID="77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9"
Oct 31 00:42:02.519109 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:02.518987    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9\": container with ID starting with 77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9 not found: ID does not exist" containerID="77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9"
Oct 31 00:42:02.519109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.519033    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9"} err="failed to get container status \"77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9\": rpc error: code = NotFound desc = could not find container \"77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9\": container with ID starting with 77ffccd456c50c3241a60d04b57761f3c7296a28bd73c007f94362a047fe8df9 not found: ID does not exist"
Oct 31 00:42:02.672708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.672660    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"]
Oct 31 00:42:02.676844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.676818    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-97fc39-647d85754d-d552c"]
Oct 31 00:42:02.690362 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.690339    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:42:02.780230 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.780143    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585\" (UniqueName: \"kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585\") pod \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") "
Oct 31 00:42:02.780407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.780242    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume\") pod \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\" (UID: \"c6a605fe-550c-40c0-b1cc-ded0e41a02b1\") "
Oct 31 00:42:02.780649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.780621    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume" pod "c6a605fe-550c-40c0-b1cc-ded0e41a02b1" (UID: "c6a605fe-550c-40c0-b1cc-ded0e41a02b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:02.782485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.782462    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585" pod "c6a605fe-550c-40c0-b1cc-ded0e41a02b1" (UID: "c6a605fe-550c-40c0-b1cc-ded0e41a02b1"). InnerVolumeSpecName "kube-api-access-hm585". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:02.881238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.881208    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm585\" (UniqueName: \"kubernetes.io/projected/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-kube-api-access-hm585\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:02.881238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:02.881236    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6a605fe-550c-40c0-b1cc-ded0e41a02b1-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:03.117923 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.117847    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ab8f75-f762-4a19-aedc-51919c61766a" path="/var/lib/kubelet/pods/30ab8f75-f762-4a19-aedc-51919c61766a/volumes"
Oct 31 00:42:03.505236 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.505198    4478 generic.go:356] "Generic (PLEG): container finished" podID="c6a605fe-550c-40c0-b1cc-ded0e41a02b1" containerID="8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95" exitCode=0
Oct 31 00:42:03.505480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.505238    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw" event={"ID":"c6a605fe-550c-40c0-b1cc-ded0e41a02b1","Type":"ContainerDied","Data":"8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95"}
Oct 31 00:42:03.505480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.505261    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"
Oct 31 00:42:03.505480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.505294    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw" event={"ID":"c6a605fe-550c-40c0-b1cc-ded0e41a02b1","Type":"ContainerDied","Data":"9d9047cd3a0f03bcc0c37989c32eb71505b98b28ac9916211106ea7d6a946f17"}
Oct 31 00:42:03.505480 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.505322    4478 scope.go:122] "RemoveContainer" containerID="8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95"
Oct 31 00:42:03.514469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.514452    4478 scope.go:122] "RemoveContainer" containerID="8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95"
Oct 31 00:42:03.514724 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:03.514693    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95\": container with ID starting with 8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95 not found: ID does not exist" containerID="8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95"
Oct 31 00:42:03.514783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.514720    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95"} err="failed to get container status \"8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95\": rpc error: code = NotFound desc = could not find container \"8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95\": container with ID starting with 8347f5362978124a8646869a2e0b81239586f05d98cdce83893119ef47fa5e95 not found: ID does not exist"
Oct 31 00:42:03.679403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.679100    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"]
Oct 31 00:42:03.683082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:03.683050    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-97fc39-79cbf8f648-l5gcw"]
Oct 31 00:42:05.118005 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:05.117962    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6a605fe-550c-40c0-b1cc-ded0e41a02b1" path="/var/lib/kubelet/pods/c6a605fe-550c-40c0-b1cc-ded0e41a02b1/volumes"
Oct 31 00:42:11.051737 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:11.051698    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"]
Oct 31 00:42:11.052199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:11.051925    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy" containerID="cri-o://be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81" gracePeriod=30
Oct 31 00:42:14.760110 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:14.760073    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:42:16.316146 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.316086    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:42:16.443720 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.443682    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.443918 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.443789    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.443918 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.443828    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.443918 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.443891    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.444082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.443938    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.444082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.443990    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.444082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.444029    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.444082 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.444078    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.444291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.444179    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs\") pod \"b36bef42-9e41-4da7-a998-103b93a9e39c\" (UID: \"b36bef42-9e41-4da7-a998-103b93a9e39c\") "
Oct 31 00:42:16.445671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.445640    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:42:16.445947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.445901    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:16.446046 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.445992    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:16.446169 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.446149    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:16.446244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.446206    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:16.447538 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.447498    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:16.448508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.448482    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:42:16.448962 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.448939    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "kube-api-access-46h2q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:42:16.450525 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.450494    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo" pod "b36bef42-9e41-4da7-a998-103b93a9e39c" (UID: "b36bef42-9e41-4da7-a998-103b93a9e39c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:42:16.545904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545875    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.545904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545903    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46h2q\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-kube-api-access-46h2q\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545914    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545924    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b36bef42-9e41-4da7-a998-103b93a9e39c-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545933    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545941    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545949    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545957    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.546075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.545965    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b36bef42-9e41-4da7-a998-103b93a9e39c-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:42:16.606357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.606280    4478 generic.go:356] "Generic (PLEG): container finished" podID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerID="be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81" exitCode=0
Oct 31 00:42:16.606357 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.606334    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"
Oct 31 00:42:16.606590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.606367    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" event={"ID":"b36bef42-9e41-4da7-a998-103b93a9e39c","Type":"ContainerDied","Data":"be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81"}
Oct 31 00:42:16.606590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.606441    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92" event={"ID":"b36bef42-9e41-4da7-a998-103b93a9e39c","Type":"ContainerDied","Data":"6779c5260fee53a38bc76aae0424ed9ea827cf48675811ba6ed578d0def96a94"}
Oct 31 00:42:16.606590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.606464    4478 scope.go:122] "RemoveContainer" containerID="be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81"
Oct 31 00:42:16.618069 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.618046    4478 scope.go:122] "RemoveContainer" containerID="263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"
Oct 31 00:42:16.626560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.626534    4478 scope.go:122] "RemoveContainer" containerID="be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81"
Oct 31 00:42:16.626830 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:16.626801    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81\": container with ID starting with be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81 not found: ID does not exist" containerID="be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81"
Oct 31 00:42:16.626898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.626837    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81"} err="failed to get container status \"be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81\": rpc error: code = NotFound desc = could not find container \"be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81\": container with ID starting with be5f554c886531c91f02e6d784cf5d7359d4d8ba3c8d5e99a18d071e20269c81 not found: ID does not exist"
Oct 31 00:42:16.626898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.626854    4478 scope.go:122] "RemoveContainer" containerID="263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"
Oct 31 00:42:16.627105 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:16.627083    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef\": container with ID starting with 263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef not found: ID does not exist" containerID="263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"
Oct 31 00:42:16.627166 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.627114    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef"} err="failed to get container status \"263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef\": rpc error: code = NotFound desc = could not find container \"263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef\": container with ID starting with 263c9bfd2f276612aa6c8143a3dc43e9bcc518c93a1c391f68e19397e97e58ef not found: ID does not exist"
Oct 31 00:42:16.658003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.657974    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"]
Oct 31 00:42:16.658637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658617    4478 cpu_manager.go:414] "containerMap: removing container" podUID="7eefc116-ddc5-4430-8833-a87ac24c378e" containerName="payload-processing"
Oct 31 00:42:16.658637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658636    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7eefc116-ddc5-4430-8833-a87ac24c378e" containerName="payload-processing"
Oct 31 00:42:16.658741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658654    4478 cpu_manager.go:414] "containerMap: removing container" podUID="30ab8f75-f762-4a19-aedc-51919c61766a" containerName="maas-api"
Oct 31 00:42:16.658741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658660    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ab8f75-f762-4a19-aedc-51919c61766a" containerName="maas-api"
Oct 31 00:42:16.658741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658670    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c6a605fe-550c-40c0-b1cc-ded0e41a02b1" containerName="payload-processing"
Oct 31 00:42:16.658741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658676    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6a605fe-550c-40c0-b1cc-ded0e41a02b1" containerName="payload-processing"
Oct 31 00:42:16.658741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658703    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy"
Oct 31 00:42:16.658741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658708    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658767    4478 cpu_manager.go:414] "containerMap: removing container" podUID="3836f164-7138-4179-8121-44b930ad8eab" containerName="payload-pre-processing"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658774    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3836f164-7138-4179-8121-44b930ad8eab" containerName="payload-pre-processing"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658869    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="7eefc116-ddc5-4430-8833-a87ac24c378e" containerName="payload-processing"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658880    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ab8f75-f762-4a19-aedc-51919c61766a" containerName="maas-api"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658886    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6a605fe-550c-40c0-b1cc-ded0e41a02b1" containerName="payload-processing"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658893    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658900    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy"
Oct 31 00:42:16.659024 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.658906    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="3836f164-7138-4179-8121-44b930ad8eab" containerName="payload-pre-processing"
Oct 31 00:42:16.663440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.663422    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.719622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.719587    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2c02a9\""
Oct 31 00:42:16.719622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.719608    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2c02a9-dockercfg-ghbd7\""
Oct 31 00:42:16.724097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.724072    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"]
Oct 31 00:42:16.748273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.748249    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mw46\" (UniqueName: \"kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.748445 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.748343    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.773273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.773251    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"]
Oct 31 00:42:16.776963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.776941    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-97fc39-openshift-default-758f6d5565-fsf92"]
Oct 31 00:42:16.850087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.850061    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.850187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.850173    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mw46\" (UniqueName: \"kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.850767 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.850749    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.915196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.915137    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mw46\" (UniqueName: \"kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46\") pod \"payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:16.977529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:16.977492    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:17.118835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.118797    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" path="/var/lib/kubelet/pods/b36bef42-9e41-4da7-a998-103b93a9e39c/volumes"
Oct 31 00:42:17.362053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.362024    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"]
Oct 31 00:42:17.369707 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.369682    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"]
Oct 31 00:42:17.370508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.370486    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy"
Oct 31 00:42:17.370508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.370507    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b36bef42-9e41-4da7-a998-103b93a9e39c" containerName="istio-proxy"
Oct 31 00:42:17.375961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.375749    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.428272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.428248    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"]
Oct 31 00:42:17.456693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.456664    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zblqj\" (UniqueName: \"kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj\") pod \"payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.456830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.456739    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.558136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.558096    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.558337 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.558296    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zblqj\" (UniqueName: \"kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj\") pod \"payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.558874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.558851    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.613723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.613648    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls" event={"ID":"14330d5b-2f53-475e-ae1f-7304f3d22b76","Type":"ContainerStarted","Data":"9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54"}
Oct 31 00:42:17.613723 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.613705    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls" event={"ID":"14330d5b-2f53-475e-ae1f-7304f3d22b76","Type":"ContainerStarted","Data":"1db5d6efc3a0180363216db67d04aab803753c46ed8833ad6a7ecf7e0b0ba72a"}
Oct 31 00:42:17.613928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.613730    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:17.619595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.619575    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zblqj\" (UniqueName: \"kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj\") pod \"payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.712961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.712930    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:17.739694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:17.739646    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls" podStartSLOduration=1.7396336209999999 podStartE2EDuration="1.739633621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:17.737689906 +0000 UTC m=+2370.754354620" watchObservedRunningTime="2026-10-31 00:42:17.739633621 +0000 UTC m=+2370.756298333"
Oct 31 00:42:18.107165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:18.107143    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"]
Oct 31 00:42:18.109605 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:42:18.109572    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14597041_78ec_4046_be80_8fc46aecfe92.slice/crio-c8e7498d2fa901c5d6d6d50532f22076defeddd35e378de33b38b68b605e2630 WatchSource:0}: Error finding container c8e7498d2fa901c5d6d6d50532f22076defeddd35e378de33b38b68b605e2630: Status 404 returned error can't find the container with id c8e7498d2fa901c5d6d6d50532f22076defeddd35e378de33b38b68b605e2630
Oct 31 00:42:18.621994 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:18.621954    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b" event={"ID":"14597041-78ec-4046-be80-8fc46aecfe92","Type":"ContainerStarted","Data":"a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42"}
Oct 31 00:42:18.622471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:18.622019    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b" event={"ID":"14597041-78ec-4046-be80-8fc46aecfe92","Type":"ContainerStarted","Data":"c8e7498d2fa901c5d6d6d50532f22076defeddd35e378de33b38b68b605e2630"}
Oct 31 00:42:18.622471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:18.622262    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:18.743692 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:18.743631    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b" podStartSLOduration=1.743613008 podStartE2EDuration="1.743613008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:18.74179689 +0000 UTC m=+2371.758461620" watchObservedRunningTime="2026-10-31 00:42:18.743613008 +0000 UTC m=+2371.760277722"
Oct 31 00:42:28.623953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:28.623875    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:42:29.631076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:29.631046    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:42:32.609772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.609739    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"]
Oct 31 00:42:32.614023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.614006    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.654456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.654431    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjsjc\" (UniqueName: \"kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.654556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.654485    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.654666 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.654645    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.654726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.654684    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.670564 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.670543    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-868b06-serving-cert\""
Oct 31 00:42:32.677546 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.677520    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"]
Oct 31 00:42:32.756123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.756092    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjsjc\" (UniqueName: \"kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.756287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.756158    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.756287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.756268    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.756449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.756306    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.756515 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:32.756462    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-serving-cert: secret "maas-api-e2e-praxis-b-868b06-serving-cert" not found
Oct 31 00:42:32.756570 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:32.756548    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls podName:dd8de6ff-a085-482b-82d4-b95095092e43 nodeName:}" failed. No retries permitted until 2026-10-31 00:42:33.256529114 +0000 UTC m=+2386.273193827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls") pod "maas-api-e2e-praxis-b-868b06-945964778-x958d" (UID: "dd8de6ff-a085-482b-82d4-b95095092e43") : secret "maas-api-e2e-praxis-b-868b06-serving-cert" not found
Oct 31 00:42:32.757217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.757143    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.759669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.759648    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:32.816934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:32.816912    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjsjc\" (UniqueName: \"kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:33.260887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:33.260858    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:33.263458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:33.263430    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\") pod \"maas-api-e2e-praxis-b-868b06-945964778-x958d\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:33.527107 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:33.527027    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:33.905036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:33.904964    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"]
Oct 31 00:42:33.908431 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:42:33.908400    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd8de6ff_a085_482b_82d4_b95095092e43.slice/crio-5f0cd6806b4ca6f17b8cd0820e02c77067d06ad3cd78e0bd11eab9d757c17f88 WatchSource:0}: Error finding container 5f0cd6806b4ca6f17b8cd0820e02c77067d06ad3cd78e0bd11eab9d757c17f88: Status 404 returned error can't find the container with id 5f0cd6806b4ca6f17b8cd0820e02c77067d06ad3cd78e0bd11eab9d757c17f88
Oct 31 00:42:33.909810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:33.909788    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:42:34.718012 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:34.717972    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d" event={"ID":"dd8de6ff-a085-482b-82d4-b95095092e43","Type":"ContainerStarted","Data":"5f0cd6806b4ca6f17b8cd0820e02c77067d06ad3cd78e0bd11eab9d757c17f88"}
Oct 31 00:42:35.739276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:35.738756    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d" event={"ID":"dd8de6ff-a085-482b-82d4-b95095092e43","Type":"ContainerStarted","Data":"f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b"}
Oct 31 00:42:35.739776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:35.739406    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:35.866629 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:35.866581    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d" podStartSLOduration=2.969345147 podStartE2EDuration="3.866566919s" totalImagesPullingTime="897.221772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:35.864461661 +0000 UTC m=+2388.881126391" watchObservedRunningTime="2026-10-31 00:42:35.866566919 +0000 UTC m=+2388.883231631"
Oct 31 00:42:41.750772 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:41.750738    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:42:44.757903 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:42:44.757862    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:42:45.725325 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.725290    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"]
Oct 31 00:42:45.729068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.729051    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:45.739421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.739391    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:45.739558 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.739544    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sznqs\" (UniqueName: \"kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs\") pod \"payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:45.788114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.788089    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"]
Oct 31 00:42:45.833013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.832985    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1a4f30\""
Oct 31 00:42:45.833785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.833764    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1a4f30-dockercfg-6hwbh\""
Oct 31 00:42:45.840313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.840291    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sznqs\" (UniqueName: \"kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs\") pod \"payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:45.840479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.840371    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:45.841060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.841039    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:45.906890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:45.906856    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sznqs\" (UniqueName: \"kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs\") pod \"payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:46.044896 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:46.044806    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:46.447090 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:46.447050    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"]
Oct 31 00:42:46.452209 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:42:46.452180    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ce33a5_0f8d_4fc6_be43_9fa34f996a54.slice/crio-9d52f6613cd762384b0ed9d219364af3d454dff52792982ae50cb42011f5446b WatchSource:0}: Error finding container 9d52f6613cd762384b0ed9d219364af3d454dff52792982ae50cb42011f5446b: Status 404 returned error can't find the container with id 9d52f6613cd762384b0ed9d219364af3d454dff52792982ae50cb42011f5446b
Oct 31 00:42:46.810176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:46.810086    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll" event={"ID":"19ce33a5-0f8d-4fc6-be43-9fa34f996a54","Type":"ContainerStarted","Data":"b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30"}
Oct 31 00:42:46.810176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:46.810146    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll" event={"ID":"19ce33a5-0f8d-4fc6-be43-9fa34f996a54","Type":"ContainerStarted","Data":"9d52f6613cd762384b0ed9d219364af3d454dff52792982ae50cb42011f5446b"}
Oct 31 00:42:46.810627 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:46.810298    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:42:46.946490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:46.946420    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll" podStartSLOduration=1.9464054160000002 podStartE2EDuration="1.946405416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:46.942867997 +0000 UTC m=+2399.959532728" watchObservedRunningTime="2026-10-31 00:42:46.946405416 +0000 UTC m=+2399.963070124"
Oct 31 00:42:57.817809 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:42:57.817781    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:43:06.234813 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:06.234767    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"]
Oct 31 00:43:06.235215 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:06.235058    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" containerID="cri-o://03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8" gracePeriod=30
Oct 31 00:43:09.303911 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:09.303861    4478 patch_prober.go:29] interesting pod/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:43:09.304427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:09.303931    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:43:11.499764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.499711    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:43:11.588102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588072    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588245 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588118    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588245 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588137    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588245 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588210    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588252    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588285    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588370    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588444    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:11.588614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588456    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588599    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data\") pod \"079636e4-f2e7-4c52-8bc7-880d0149f24c\" (UID: \"079636e4-f2e7-4c52-8bc7-880d0149f24c\") "
Oct 31 00:43:11.588764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588615    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:11.588764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588636    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:11.588764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588699    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:11.588931 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.588885    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:11.589077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.589055    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.589077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.589080    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.589077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.589094    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.589447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.589109    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.589447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.589121    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/079636e4-f2e7-4c52-8bc7-880d0149f24c-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.590659 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.590633    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:11.590827 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.590802    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "kube-api-access-b542c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:11.591083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.591056    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:43:11.591083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.591061    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token" pod "079636e4-f2e7-4c52-8bc7-880d0149f24c" (UID: "079636e4-f2e7-4c52-8bc7-880d0149f24c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:11.690835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.690806    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b542c\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-kube-api-access-b542c\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.690835 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.690834    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.691041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.690845    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.691041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.690857    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/079636e4-f2e7-4c52-8bc7-880d0149f24c-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:11.966682 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.966647    4478 generic.go:356] "Generic (PLEG): container finished" podID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerID="03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8" exitCode=0
Oct 31 00:43:11.966871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.966721    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"
Oct 31 00:43:11.966871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.966740    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" event={"ID":"079636e4-f2e7-4c52-8bc7-880d0149f24c","Type":"ContainerDied","Data":"03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8"}
Oct 31 00:43:11.966871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.966817    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp" event={"ID":"079636e4-f2e7-4c52-8bc7-880d0149f24c","Type":"ContainerDied","Data":"c08fd6e54793d06569fdfa30f1369a24c9f9f75032a539747fbdc2903ae99c66"}
Oct 31 00:43:11.966871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.966850    4478 scope.go:122] "RemoveContainer" containerID="03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8"
Oct 31 00:43:11.979038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.979022    4478 scope.go:122] "RemoveContainer" containerID="edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"
Oct 31 00:43:11.988675 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.988655    4478 scope.go:122] "RemoveContainer" containerID="03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8"
Oct 31 00:43:11.988930 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:11.988912    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8\": container with ID starting with 03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8 not found: ID does not exist" containerID="03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8"
Oct 31 00:43:11.989003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.988944    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8"} err="failed to get container status \"03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8\": rpc error: code = NotFound desc = could not find container \"03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8\": container with ID starting with 03e539f0d0b5f0a564428eb631959b0b94d116cae944ee080e779122475c11b8 not found: ID does not exist"
Oct 31 00:43:11.989003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.988968    4478 scope.go:122] "RemoveContainer" containerID="edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"
Oct 31 00:43:11.989219 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:11.989202    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c\": container with ID starting with edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c not found: ID does not exist" containerID="edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"
Oct 31 00:43:11.989268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:11.989226    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c"} err="failed to get container status \"edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c\": rpc error: code = NotFound desc = could not find container \"edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c\": container with ID starting with edbf8c657d43177228b9d02d119498d89f29c7fea949b16607abe2e81b87b57c not found: ID does not exist"
Oct 31 00:43:12.133817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:12.133787    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"]
Oct 31 00:43:12.184065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:12.184036    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-752be8-openshift-default-79688d7695-mvqqp"]
Oct 31 00:43:13.118649 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.118608    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" path="/var/lib/kubelet/pods/079636e4-f2e7-4c52-8bc7-880d0149f24c/volumes"
Oct 31 00:43:13.640837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.640802    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"]
Oct 31 00:43:13.641044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.641018    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc" podUID="1ab6d036-7c66-488f-a5e3-05f3bc47bab3" containerName="maas-api" containerID="cri-o://f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7" gracePeriod=30
Oct 31 00:43:13.942088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.942054    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:43:13.980934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.980905    4478 generic.go:356] "Generic (PLEG): container finished" podID="1ab6d036-7c66-488f-a5e3-05f3bc47bab3" containerID="f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7" exitCode=0
Oct 31 00:43:13.981075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.980959    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"
Oct 31 00:43:13.981075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.980997    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc" event={"ID":"1ab6d036-7c66-488f-a5e3-05f3bc47bab3","Type":"ContainerDied","Data":"f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7"}
Oct 31 00:43:13.981075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.981058    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc" event={"ID":"1ab6d036-7c66-488f-a5e3-05f3bc47bab3","Type":"ContainerDied","Data":"f8dbbe6ed582d811215cd77e51cfcecf23d554618877791181713b04e88ecc6b"}
Oct 31 00:43:13.981075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.981071    4478 scope.go:122] "RemoveContainer" containerID="f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7"
Oct 31 00:43:13.990226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.990207    4478 scope.go:122] "RemoveContainer" containerID="f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7"
Oct 31 00:43:13.990531 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:13.990510    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7\": container with ID starting with f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7 not found: ID does not exist" containerID="f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7"
Oct 31 00:43:13.990605 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:13.990542    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7"} err="failed to get container status \"f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7\": rpc error: code = NotFound desc = could not find container \"f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7\": container with ID starting with f5c3f84e8c9121533f6abcd3523877c8a29a545d945be8af7e72f08944b9d3b7 not found: ID does not exist"
Oct 31 00:43:14.117115 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.117077    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh\" (UniqueName: \"kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh\") pod \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") "
Oct 31 00:43:14.117302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.117156    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca\") pod \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") "
Oct 31 00:43:14.117302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.117219    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls\") pod \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") "
Oct 31 00:43:14.117458 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.117346    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs\") pod \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\" (UID: \"1ab6d036-7c66-488f-a5e3-05f3bc47bab3\") "
Oct 31 00:43:14.117620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.117589    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca" pod "1ab6d036-7c66-488f-a5e3-05f3bc47bab3" (UID: "1ab6d036-7c66-488f-a5e3-05f3bc47bab3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:14.117829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.117811    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:14.120148 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.120125    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh" pod "1ab6d036-7c66-488f-a5e3-05f3bc47bab3" (UID: "1ab6d036-7c66-488f-a5e3-05f3bc47bab3"). InnerVolumeSpecName "kube-api-access-5xqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:14.120542 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.120519    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls" pod "1ab6d036-7c66-488f-a5e3-05f3bc47bab3" (UID: "1ab6d036-7c66-488f-a5e3-05f3bc47bab3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:43:14.121684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.121666    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs" pod "1ab6d036-7c66-488f-a5e3-05f3bc47bab3" (UID: "1ab6d036-7c66-488f-a5e3-05f3bc47bab3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:43:14.219659 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.219320    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:14.219659 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.219356    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xqmh\" (UniqueName: \"kubernetes.io/projected/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-kube-api-access-5xqmh\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:14.219659 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.219374    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ab6d036-7c66-488f-a5e3-05f3bc47bab3-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:14.403003 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.402972    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"]
Oct 31 00:43:14.408631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:14.408605    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-f6a8a5-bf9d96d84-v6tlc"]
Oct 31 00:43:14.762622 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:14.762584    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:43:15.117319 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.117241    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab6d036-7c66-488f-a5e3-05f3bc47bab3" path="/var/lib/kubelet/pods/1ab6d036-7c66-488f-a5e3-05f3bc47bab3/volumes"
Oct 31 00:43:15.555559 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.555516    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"]
Oct 31 00:43:15.556054 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.555773    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b" podUID="14597041-78ec-4046-be80-8fc46aecfe92" containerName="payload-processing" containerID="cri-o://a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42" gracePeriod=30
Oct 31 00:43:15.612285 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.612166    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"]
Oct 31 00:43:15.613185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.612863    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls" podUID="14330d5b-2f53-475e-ae1f-7304f3d22b76" containerName="payload-pre-processing" containerID="cri-o://9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54" gracePeriod=30
Oct 31 00:43:15.908625 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.908589    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:43:15.932987 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.932965    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:43:15.959878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.959851    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume\") pod \"14597041-78ec-4046-be80-8fc46aecfe92\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") "
Oct 31 00:43:15.960034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.959926    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj\" (UniqueName: \"kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj\") pod \"14597041-78ec-4046-be80-8fc46aecfe92\" (UID: \"14597041-78ec-4046-be80-8fc46aecfe92\") "
Oct 31 00:43:15.960259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.960233    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume" pod "14597041-78ec-4046-be80-8fc46aecfe92" (UID: "14597041-78ec-4046-be80-8fc46aecfe92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:15.962272 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.962249    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj" pod "14597041-78ec-4046-be80-8fc46aecfe92" (UID: "14597041-78ec-4046-be80-8fc46aecfe92"). InnerVolumeSpecName "kube-api-access-zblqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:15.993734 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.993709    4478 generic.go:356] "Generic (PLEG): container finished" podID="14597041-78ec-4046-be80-8fc46aecfe92" containerID="a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42" exitCode=0
Oct 31 00:43:15.993844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.993744    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b" event={"ID":"14597041-78ec-4046-be80-8fc46aecfe92","Type":"ContainerDied","Data":"a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42"}
Oct 31 00:43:15.993844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.993806    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b" event={"ID":"14597041-78ec-4046-be80-8fc46aecfe92","Type":"ContainerDied","Data":"c8e7498d2fa901c5d6d6d50532f22076defeddd35e378de33b38b68b605e2630"}
Oct 31 00:43:15.993844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.993756    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"
Oct 31 00:43:15.993844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.993835    4478 scope.go:122] "RemoveContainer" containerID="a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42"
Oct 31 00:43:15.995221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.995199    4478 generic.go:356] "Generic (PLEG): container finished" podID="14330d5b-2f53-475e-ae1f-7304f3d22b76" containerID="9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54" exitCode=0
Oct 31 00:43:15.995324 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.995245    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"
Oct 31 00:43:15.995324 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.995290    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls" event={"ID":"14330d5b-2f53-475e-ae1f-7304f3d22b76","Type":"ContainerDied","Data":"9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54"}
Oct 31 00:43:15.995439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:15.995340    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls" event={"ID":"14330d5b-2f53-475e-ae1f-7304f3d22b76","Type":"ContainerDied","Data":"1db5d6efc3a0180363216db67d04aab803753c46ed8833ad6a7ecf7e0b0ba72a"}
Oct 31 00:43:16.003708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.003691    4478 scope.go:122] "RemoveContainer" containerID="a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42"
Oct 31 00:43:16.003961 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:16.003943    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42\": container with ID starting with a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42 not found: ID does not exist" containerID="a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42"
Oct 31 00:43:16.004031 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.003973    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42"} err="failed to get container status \"a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42\": rpc error: code = NotFound desc = could not find container \"a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42\": container with ID starting with a90592a8b49a1957d6fdbe4d3c22fc14cf692cf3fd3b8ac15ab41a2b5b127a42 not found: ID does not exist"
Oct 31 00:43:16.004031 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.003996    4478 scope.go:122] "RemoveContainer" containerID="9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54"
Oct 31 00:43:16.012108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.012091    4478 scope.go:122] "RemoveContainer" containerID="9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54"
Oct 31 00:43:16.012345 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:16.012326    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54\": container with ID starting with 9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54 not found: ID does not exist" containerID="9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54"
Oct 31 00:43:16.012415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.012355    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54"} err="failed to get container status \"9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54\": rpc error: code = NotFound desc = could not find container \"9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54\": container with ID starting with 9c9ce63ef10a8d3b44c5deb942bc2e38b9e6ae1366a2194a4775297498edad54 not found: ID does not exist"
Oct 31 00:43:16.061467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.061432    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume\") pod \"14330d5b-2f53-475e-ae1f-7304f3d22b76\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") "
Oct 31 00:43:16.061620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.061555    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46\" (UniqueName: \"kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46\") pod \"14330d5b-2f53-475e-ae1f-7304f3d22b76\" (UID: \"14330d5b-2f53-475e-ae1f-7304f3d22b76\") "
Oct 31 00:43:16.061865 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.061837    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume" pod "14330d5b-2f53-475e-ae1f-7304f3d22b76" (UID: "14330d5b-2f53-475e-ae1f-7304f3d22b76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:16.062359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.062177    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14330d5b-2f53-475e-ae1f-7304f3d22b76-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:16.062359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.062202    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14597041-78ec-4046-be80-8fc46aecfe92-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:16.062359 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.062217    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zblqj\" (UniqueName: \"kubernetes.io/projected/14597041-78ec-4046-be80-8fc46aecfe92-kube-api-access-zblqj\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:16.064821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.064792    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46" pod "14330d5b-2f53-475e-ae1f-7304f3d22b76" (UID: "14330d5b-2f53-475e-ae1f-7304f3d22b76"). InnerVolumeSpecName "kube-api-access-2mw46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:16.161476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.160711    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"]
Oct 31 00:43:16.165406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.165337    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2c02a9-795bf5b9fd-rdd6b"]
Oct 31 00:43:16.174840 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.174787    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mw46\" (UniqueName: \"kubernetes.io/projected/14330d5b-2f53-475e-ae1f-7304f3d22b76-kube-api-access-2mw46\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:16.419882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.419799    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"]
Oct 31 00:43:16.427875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:16.427851    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2c02a9-6b44499ff9-lrvls"]
Oct 31 00:43:17.119791 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:17.119751    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14330d5b-2f53-475e-ae1f-7304f3d22b76" path="/var/lib/kubelet/pods/14330d5b-2f53-475e-ae1f-7304f3d22b76/volumes"
Oct 31 00:43:17.120266 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:17.120249    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14597041-78ec-4046-be80-8fc46aecfe92" path="/var/lib/kubelet/pods/14597041-78ec-4046-be80-8fc46aecfe92/volumes"
Oct 31 00:43:18.910505 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:18.910472    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"]
Oct 31 00:43:23.911353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:23.911308    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t" podUID="14cd784f-7ce6-430c-b879-d4b7c4b2f89a" containerName="payload-processing" containerID="cri-o://1a537948f761847c9b5cc051b21e1cb9ecea50d5dc33d697693f89bc669dc996" gracePeriod=25
Oct 31 00:43:24.050601 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.050564    4478 generic.go:356] "Generic (PLEG): container finished" podID="14cd784f-7ce6-430c-b879-d4b7c4b2f89a" containerID="1a537948f761847c9b5cc051b21e1cb9ecea50d5dc33d697693f89bc669dc996" exitCode=0
Oct 31 00:43:24.050751 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.050642    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t" event={"ID":"14cd784f-7ce6-430c-b879-d4b7c4b2f89a","Type":"ContainerDied","Data":"1a537948f761847c9b5cc051b21e1cb9ecea50d5dc33d697693f89bc669dc996"}
Oct 31 00:43:24.171619 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.171550    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:43:24.273026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.272991    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6\" (UniqueName: \"kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6\") pod \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") "
Oct 31 00:43:24.273196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.273053    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume\") pod \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\" (UID: \"14cd784f-7ce6-430c-b879-d4b7c4b2f89a\") "
Oct 31 00:43:24.273522 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.273491    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume" pod "14cd784f-7ce6-430c-b879-d4b7c4b2f89a" (UID: "14cd784f-7ce6-430c-b879-d4b7c4b2f89a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:24.275440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.275417    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6" pod "14cd784f-7ce6-430c-b879-d4b7c4b2f89a" (UID: "14cd784f-7ce6-430c-b879-d4b7c4b2f89a"). InnerVolumeSpecName "kube-api-access-4g8z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:24.374741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.374709    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g8z6\" (UniqueName: \"kubernetes.io/projected/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-kube-api-access-4g8z6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:24.374741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:24.374739    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14cd784f-7ce6-430c-b879-d4b7c4b2f89a-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:25.058073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:25.058045    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"
Oct 31 00:43:25.058484 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:25.058068    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t" event={"ID":"14cd784f-7ce6-430c-b879-d4b7c4b2f89a","Type":"ContainerDied","Data":"a95c80bbae7b414911e4690a74f08254398344fbc18a06fa98ff730592fa0d7e"}
Oct 31 00:43:25.058484 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:25.058147    4478 scope.go:122] "RemoveContainer" containerID="1a537948f761847c9b5cc051b21e1cb9ecea50d5dc33d697693f89bc669dc996"
Oct 31 00:43:25.242320 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:25.242285    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"]
Oct 31 00:43:25.291327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:25.291296    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-726d5b-bb84b569c-9wn4t"]
Oct 31 00:43:27.118890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:27.118849    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cd784f-7ce6-430c-b879-d4b7c4b2f89a" path="/var/lib/kubelet/pods/14cd784f-7ce6-430c-b879-d4b7c4b2f89a/volumes"
Oct 31 00:43:38.355631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.355596    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"]
Oct 31 00:43:38.356103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.355815    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll" podUID="19ce33a5-0f8d-4fc6-be43-9fa34f996a54" containerName="payload-processing" containerID="cri-o://b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30" gracePeriod=30
Oct 31 00:43:38.672180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.671860    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:43:38.784647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.784615    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume\") pod \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") "
Oct 31 00:43:38.784833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.784818    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs\" (UniqueName: \"kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs\") pod \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\" (UID: \"19ce33a5-0f8d-4fc6-be43-9fa34f996a54\") "
Oct 31 00:43:38.785028 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.784999    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume" pod "19ce33a5-0f8d-4fc6-be43-9fa34f996a54" (UID: "19ce33a5-0f8d-4fc6-be43-9fa34f996a54"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:38.785239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.785215    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:38.787324 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.787294    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs" pod "19ce33a5-0f8d-4fc6-be43-9fa34f996a54" (UID: "19ce33a5-0f8d-4fc6-be43-9fa34f996a54"). InnerVolumeSpecName "kube-api-access-sznqs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:38.886703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:38.886627    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sznqs\" (UniqueName: \"kubernetes.io/projected/19ce33a5-0f8d-4fc6-be43-9fa34f996a54-kube-api-access-sznqs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:39.270304 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.270269    4478 generic.go:356] "Generic (PLEG): container finished" podID="19ce33a5-0f8d-4fc6-be43-9fa34f996a54" containerID="b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30" exitCode=0
Oct 31 00:43:39.270513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.270330    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"
Oct 31 00:43:39.270513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.270372    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll" event={"ID":"19ce33a5-0f8d-4fc6-be43-9fa34f996a54","Type":"ContainerDied","Data":"b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30"}
Oct 31 00:43:39.270513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.270470    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll" event={"ID":"19ce33a5-0f8d-4fc6-be43-9fa34f996a54","Type":"ContainerDied","Data":"9d52f6613cd762384b0ed9d219364af3d454dff52792982ae50cb42011f5446b"}
Oct 31 00:43:39.270513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.270500    4478 scope.go:122] "RemoveContainer" containerID="b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30"
Oct 31 00:43:39.282020 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.281999    4478 scope.go:122] "RemoveContainer" containerID="b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30"
Oct 31 00:43:39.282343 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:39.282321    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30\": container with ID starting with b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30 not found: ID does not exist" containerID="b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30"
Oct 31 00:43:39.282443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.282352    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30"} err="failed to get container status \"b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30\": rpc error: code = NotFound desc = could not find container \"b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30\": container with ID starting with b7cadc60753eb15ef532575d73800bc5ab5920dc2d34085a8d54276d6d94cf30 not found: ID does not exist"
Oct 31 00:43:39.434599 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.434564    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"]
Oct 31 00:43:39.440759 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:39.440729    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1a4f30-77b6c6c978-4vvll"]
Oct 31 00:43:41.117468 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:41.117438    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ce33a5-0f8d-4fc6-be43-9fa34f996a54" path="/var/lib/kubelet/pods/19ce33a5-0f8d-4fc6-be43-9fa34f996a54/volumes"
Oct 31 00:43:44.757657 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:44.757618    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:43:48.578127 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:48.578090    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"]
Oct 31 00:43:48.578619 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:48.578305    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy" containerID="cri-o://a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f" gracePeriod=30
Oct 31 00:43:53.872533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.872505    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:43:53.992365 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992333    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992365 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992365    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992497    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992612 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992563    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992612 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992596    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992655    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992676    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:53.992701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992691    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992804    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.992934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992838    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token\") pod \"91616be9-2019-4851-a560-5d2979f2f9b1\" (UID: \"91616be9-2019-4851-a560-5d2979f2f9b1\") "
Oct 31 00:43:53.993050 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992831    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:53.993050 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992918    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:53.993050 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.992939    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:43:53.993224 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.993213    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:53.993282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.993231    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91616be9-2019-4851-a560-5d2979f2f9b1-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:53.993282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.993247    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:53.993282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.993259    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:53.993282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.993211    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:53.995101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.995072    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:43:53.995205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.995096    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:53.995205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.995178    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:43:53.996512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:53.996488    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4" pod "91616be9-2019-4851-a560-5d2979f2f9b1" (UID: "91616be9-2019-4851-a560-5d2979f2f9b1"). InnerVolumeSpecName "kube-api-access-rvzd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:43:54.094677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.094649    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91616be9-2019-4851-a560-5d2979f2f9b1-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:54.094677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.094677    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvzd4\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-kube-api-access-rvzd4\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:54.094878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.094688    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:54.094878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.094696    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91616be9-2019-4851-a560-5d2979f2f9b1-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:54.094878 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.094707    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91616be9-2019-4851-a560-5d2979f2f9b1-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:43:54.370124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.370084    4478 generic.go:356] "Generic (PLEG): container finished" podID="91616be9-2019-4851-a560-5d2979f2f9b1" containerID="a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f" exitCode=0
Oct 31 00:43:54.370308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.370151    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"
Oct 31 00:43:54.370308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.370172    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" event={"ID":"91616be9-2019-4851-a560-5d2979f2f9b1","Type":"ContainerDied","Data":"a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f"}
Oct 31 00:43:54.370308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.370235    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn" event={"ID":"91616be9-2019-4851-a560-5d2979f2f9b1","Type":"ContainerDied","Data":"9a0bf287ff54d0d38a847841132b6dc49445e4061410018135657586b207ce89"}
Oct 31 00:43:54.370308 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.370266    4478 scope.go:122] "RemoveContainer" containerID="a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f"
Oct 31 00:43:54.383180 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.383160    4478 scope.go:122] "RemoveContainer" containerID="b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"
Oct 31 00:43:54.391238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.391220    4478 scope.go:122] "RemoveContainer" containerID="a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f"
Oct 31 00:43:54.391514 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:54.391495    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f\": container with ID starting with a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f not found: ID does not exist" containerID="a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f"
Oct 31 00:43:54.391583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.391527    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f"} err="failed to get container status \"a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f\": rpc error: code = NotFound desc = could not find container \"a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f\": container with ID starting with a18ee711c8fda7f463daa00f5ac1104e1946a491e82ab412599b4c497ee7700f not found: ID does not exist"
Oct 31 00:43:54.391583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.391552    4478 scope.go:122] "RemoveContainer" containerID="b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"
Oct 31 00:43:54.391797 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:43:54.391779    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03\": container with ID starting with b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03 not found: ID does not exist" containerID="b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"
Oct 31 00:43:54.391839 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.391804    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03"} err="failed to get container status \"b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03\": rpc error: code = NotFound desc = could not find container \"b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03\": container with ID starting with b018a97f9e8bc82fb52a6d1062eb8a8a1027dd877b206b2e1f49a505216e2a03 not found: ID does not exist"
Oct 31 00:43:54.548273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.548233    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"]
Oct 31 00:43:54.551541 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:54.551515    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-1a4f30-openshift-default-84bc5b7599-pbgcn"]
Oct 31 00:43:55.118552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:55.118522    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" path="/var/lib/kubelet/pods/91616be9-2019-4851-a560-5d2979f2f9b1/volumes"
Oct 31 00:43:59.727850 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.727772    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"]
Oct 31 00:43:59.728287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728248    4478 cpu_manager.go:414] "containerMap: removing container" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy"
Oct 31 00:43:59.728287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728259    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy"
Oct 31 00:43:59.728287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728269    4478 cpu_manager.go:414] "containerMap: removing container" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy"
Oct 31 00:43:59.728287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728274    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy"
Oct 31 00:43:59.728287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728285    4478 cpu_manager.go:414] "containerMap: removing container" podUID="1ab6d036-7c66-488f-a5e3-05f3bc47bab3" containerName="maas-api"
Oct 31 00:43:59.728287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728290    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab6d036-7c66-488f-a5e3-05f3bc47bab3" containerName="maas-api"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728296    4478 cpu_manager.go:414] "containerMap: removing container" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728301    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728311    4478 cpu_manager.go:414] "containerMap: removing container" podUID="14597041-78ec-4046-be80-8fc46aecfe92" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728315    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14597041-78ec-4046-be80-8fc46aecfe92" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728328    4478 cpu_manager.go:414] "containerMap: removing container" podUID="14cd784f-7ce6-430c-b879-d4b7c4b2f89a" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728333    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cd784f-7ce6-430c-b879-d4b7c4b2f89a" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728341    4478 cpu_manager.go:414] "containerMap: removing container" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728346    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728362    4478 cpu_manager.go:414] "containerMap: removing container" podUID="14330d5b-2f53-475e-ae1f-7304f3d22b76" containerName="payload-pre-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728367    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14330d5b-2f53-475e-ae1f-7304f3d22b76" containerName="payload-pre-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728374    4478 cpu_manager.go:414] "containerMap: removing container" podUID="19ce33a5-0f8d-4fc6-be43-9fa34f996a54" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728397    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ce33a5-0f8d-4fc6-be43-9fa34f996a54" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728477    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728485    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="91616be9-2019-4851-a560-5d2979f2f9b1" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728491    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab6d036-7c66-488f-a5e3-05f3bc47bab3" containerName="maas-api"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728496    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728501    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="14597041-78ec-4046-be80-8fc46aecfe92" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728508    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cd784f-7ce6-430c-b879-d4b7c4b2f89a" containerName="payload-processing"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728514    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="079636e4-f2e7-4c52-8bc7-880d0149f24c" containerName="istio-proxy"
Oct 31 00:43:59.728513 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728522    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="14330d5b-2f53-475e-ae1f-7304f3d22b76" containerName="payload-pre-processing"
Oct 31 00:43:59.729139 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.728527    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ce33a5-0f8d-4fc6-be43-9fa34f996a54" containerName="payload-processing"
Oct 31 00:43:59.733073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.733058    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:43:59.787573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.787548    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-6a693b\""
Oct 31 00:43:59.789043 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.789021    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:43:59.789139 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.789120    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh9pv\" (UniqueName: \"kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv\") pod \"payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:43:59.790937 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.790920    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-6a693b-dockercfg-k644v\""
Oct 31 00:43:59.844496 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.844448    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"]
Oct 31 00:43:59.918406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.915306    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:43:59.918406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.915404    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh9pv\" (UniqueName: \"kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv\") pod \"payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:43:59.918406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.916724    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:43:59.924904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.924875    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"]
Oct 31 00:43:59.929884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.929869    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:43:59.952028 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.951997    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"]
Oct 31 00:43:59.977205 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:43:59.977176    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh9pv\" (UniqueName: \"kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv\") pod \"payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:44:00.016764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.016682    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxq8\" (UniqueName: \"kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8\") pod \"payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.016764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.016759    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.048569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.048537    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:44:00.119702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.119674    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.120138 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.120111    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxq8\" (UniqueName: \"kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8\") pod \"payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.120611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.120586    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.181658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.181620    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxq8\" (UniqueName: \"kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8\") pod \"payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.246141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.246100    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:00.438640 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.438602    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"]
Oct 31 00:44:00.445971 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:44:00.445930    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15031097_609b_4830_a87b_eb2ed9494574.slice/crio-12cb0c9173d0b6603703137cc87a9114c5cc342c3f5916570ebc24df028fdb25 WatchSource:0}: Error finding container 12cb0c9173d0b6603703137cc87a9114c5cc342c3f5916570ebc24df028fdb25: Status 404 returned error can't find the container with id 12cb0c9173d0b6603703137cc87a9114c5cc342c3f5916570ebc24df028fdb25
Oct 31 00:44:00.628580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:00.628552    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"]
Oct 31 00:44:00.629843 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:44:00.629812    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64d3be45_e851_4c25_85fd_bf83a26e2bfa.slice/crio-49f0166d689b5faa3d72d78d43293b80c59c1c99b99ac74570510fc626a31166 WatchSource:0}: Error finding container 49f0166d689b5faa3d72d78d43293b80c59c1c99b99ac74570510fc626a31166: Status 404 returned error can't find the container with id 49f0166d689b5faa3d72d78d43293b80c59c1c99b99ac74570510fc626a31166
Oct 31 00:44:01.412284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.412241    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz" event={"ID":"64d3be45-e851-4c25-85fd-bf83a26e2bfa","Type":"ContainerStarted","Data":"933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c"}
Oct 31 00:44:01.412811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.412299    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz" event={"ID":"64d3be45-e851-4c25-85fd-bf83a26e2bfa","Type":"ContainerStarted","Data":"49f0166d689b5faa3d72d78d43293b80c59c1c99b99ac74570510fc626a31166"}
Oct 31 00:44:01.412811 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.412454    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:01.414026 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.414001    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts" event={"ID":"15031097-609b-4830-a87b-eb2ed9494574","Type":"ContainerStarted","Data":"c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9"}
Oct 31 00:44:01.414152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.414047    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:44:01.414152 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.414064    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts" event={"ID":"15031097-609b-4830-a87b-eb2ed9494574","Type":"ContainerStarted","Data":"12cb0c9173d0b6603703137cc87a9114c5cc342c3f5916570ebc24df028fdb25"}
Oct 31 00:44:01.537663 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.537602    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz" podStartSLOduration=2.537584731 podStartE2EDuration="2.537584731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:01.534625486 +0000 UTC m=+2474.551290216" watchObservedRunningTime="2026-10-31 00:44:01.537584731 +0000 UTC m=+2474.554249444"
Oct 31 00:44:01.668587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:01.668455    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts" podStartSLOduration=2.668433767 podStartE2EDuration="2.668433767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:01.667789469 +0000 UTC m=+2474.684454203" watchObservedRunningTime="2026-10-31 00:44:01.668433767 +0000 UTC m=+2474.685098479"
Oct 31 00:44:08.397150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:08.397116    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"]
Oct 31 00:44:08.397623 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:08.397318    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" containerID="cri-o://098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098" gracePeriod=30
Oct 31 00:44:12.064324 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:12.064289    4478 patch_prober.go:29] interesting pod/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:44:12.064708 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:12.064353    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:44:12.422783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:12.422685    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:44:12.422783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:12.422756    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:44:13.656292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.656256    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:44:13.701537 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701510    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.701671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701574    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.701671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701595    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.701671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701625    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.701671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701656    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.701888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701756    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.701888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701784    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:13.701888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701796    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.702044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701888    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.702044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701906    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:13.702044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.701941    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert\") pod \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\" (UID: \"c33bf3fd-0d13-4b7e-bb51-b11249184aba\") "
Oct 31 00:44:13.702217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702066    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:13.702217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702190    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:13.702653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702537    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:44:13.702653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702619    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.702868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702675    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.702868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702695    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.702868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.702710    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.704131 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.704105    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:44:13.704131 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.704117    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:13.704273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.704136    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:13.704273 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.704159    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h" pod "c33bf3fd-0d13-4b7e-bb51-b11249184aba" (UID: "c33bf3fd-0d13-4b7e-bb51-b11249184aba"). InnerVolumeSpecName "kube-api-access-kqv5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:13.803966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.803943    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.803966 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.803966    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.804130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.803977    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.804130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.803987    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kqv5h\" (UniqueName: \"kubernetes.io/projected/c33bf3fd-0d13-4b7e-bb51-b11249184aba-kube-api-access-kqv5h\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:13.804130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:13.803997    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c33bf3fd-0d13-4b7e-bb51-b11249184aba-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:14.485827 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.485780    4478 generic.go:356] "Generic (PLEG): container finished" podID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerID="098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098" exitCode=0
Oct 31 00:44:14.486023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.485833    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"
Oct 31 00:44:14.486023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.485908    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" event={"ID":"c33bf3fd-0d13-4b7e-bb51-b11249184aba","Type":"ContainerDied","Data":"098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098"}
Oct 31 00:44:14.486023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.485959    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g" event={"ID":"c33bf3fd-0d13-4b7e-bb51-b11249184aba","Type":"ContainerDied","Data":"a8958978c8ed9ceb48060413a3fd05c4e41dd2cf6ceb6f381ded30296a0d04b6"}
Oct 31 00:44:14.486023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.485981    4478 scope.go:122] "RemoveContainer" containerID="098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098"
Oct 31 00:44:14.496900 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.496890    4478 scope.go:122] "RemoveContainer" containerID="098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098"
Oct 31 00:44:14.497188 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:44:14.497169    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098\": container with ID starting with 098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098 not found: ID does not exist" containerID="098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098"
Oct 31 00:44:14.497260 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.497194    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098"} err="failed to get container status \"098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098\": rpc error: code = NotFound desc = could not find container \"098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098\": container with ID starting with 098dc4885eacbe314df802402a4c824688b70e9e4f64b0090dbd3d144da0f098 not found: ID does not exist"
Oct 31 00:44:14.702781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.702732    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"]
Oct 31 00:44:14.706983 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:14.706957    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-0144bd-openshift-default-67b9448dc-bdp2g"]
Oct 31 00:44:14.757198 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:44:14.757131    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:44:15.117244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:15.117169    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" path="/var/lib/kubelet/pods/c33bf3fd-0d13-4b7e-bb51-b11249184aba/volumes"
Oct 31 00:44:20.280710 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.280667    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"]
Oct 31 00:44:20.281233 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.281217    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy"
Oct 31 00:44:20.281233 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.281233    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy"
Oct 31 00:44:20.281367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.281331    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c33bf3fd-0d13-4b7e-bb51-b11249184aba" containerName="istio-proxy"
Oct 31 00:44:20.286066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.286049    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.341229 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.341196    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-7ae676-serving-cert\""
Oct 31 00:44:20.371634 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.371593    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.371821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.371746    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.371887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.371841    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.371887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.371879    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdskh\" (UniqueName: \"kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.392809 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.392778    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"]
Oct 31 00:44:20.473726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.473676    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.473895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.473769    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.473895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.473821    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.473895 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.473851    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdskh\" (UniqueName: \"kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.474071 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:44:20.473980    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-serving-cert: secret "maas-api-e2e-praxis-a-7ae676-serving-cert" not found
Oct 31 00:44:20.474071 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:44:20.474055    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls podName:cd67942f-ee48-4d55-aebf-48bd6756f49e nodeName:}" failed. No retries permitted until 2026-10-31 00:44:20.974034344 +0000 UTC m=+2493.990699055 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls") pod "maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" (UID: "cd67942f-ee48-4d55-aebf-48bd6756f49e") : secret "maas-api-e2e-praxis-a-7ae676-serving-cert" not found
Oct 31 00:44:20.474746 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.474718    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.477255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.477225    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.540529 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.540449    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdskh\" (UniqueName: \"kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.980693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.980614    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:20.983439 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:20.983413    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\") pod \"maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:21.198944 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:21.198915    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:21.587459 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:21.587432    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"]
Oct 31 00:44:21.589338 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:44:21.589305    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd67942f_ee48_4d55_aebf_48bd6756f49e.slice/crio-91a5b100b61b67ee7a17f7f4a13bb4294903580327c12619138a72fb4562789c WatchSource:0}: Error finding container 91a5b100b61b67ee7a17f7f4a13bb4294903580327c12619138a72fb4562789c: Status 404 returned error can't find the container with id 91a5b100b61b67ee7a17f7f4a13bb4294903580327c12619138a72fb4562789c
Oct 31 00:44:22.532226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:22.532190    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" event={"ID":"cd67942f-ee48-4d55-aebf-48bd6756f49e","Type":"ContainerStarted","Data":"4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc"}
Oct 31 00:44:22.532403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:22.532251    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" event={"ID":"cd67942f-ee48-4d55-aebf-48bd6756f49e","Type":"ContainerStarted","Data":"91a5b100b61b67ee7a17f7f4a13bb4294903580327c12619138a72fb4562789c"}
Oct 31 00:44:22.532403 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:22.532280    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:22.650323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:22.650259    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" podStartSLOduration=1.8371249060000001 podStartE2EDuration="2.650239996s" totalImagesPullingTime="813.11509ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:22.648582538 +0000 UTC m=+2495.665247276" watchObservedRunningTime="2026-10-31 00:44:22.650239996 +0000 UTC m=+2495.666904711"
Oct 31 00:44:25.834062 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:25.834024    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"]
Oct 31 00:44:25.834696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:25.834459    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy" containerID="cri-o://1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211" gracePeriod=30
Oct 31 00:44:28.543448 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:28.543421    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:44:31.096845 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.096821    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:44:31.206128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206102    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206135    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206185    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206204    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206227    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206294 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206282    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206348    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206434    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206498    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy\") pod \"da1c2756-594d-4018-a886-260b76006523\" (UID: \"da1c2756-594d-4018-a886-260b76006523\") "
Oct 31 00:44:31.206607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206532    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:31.206607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206558    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:31.206954 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206608    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:31.207013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206961    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.207013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206969    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:31.207013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206982    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.207013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.206997    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.207323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.207137    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:44:31.208594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.208573    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:31.208809 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.208792    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:44:31.208881 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.208835    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:44:31.208995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.208976    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j" pod "da1c2756-594d-4018-a886-260b76006523" (UID: "da1c2756-594d-4018-a886-260b76006523"). InnerVolumeSpecName "kube-api-access-bxn5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:44:31.308600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.308566    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da1c2756-594d-4018-a886-260b76006523-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.308600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.308594    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.308600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.308604    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxn5j\" (UniqueName: \"kubernetes.io/projected/da1c2756-594d-4018-a886-260b76006523-kube-api-access-bxn5j\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.308810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.308615    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.308810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.308625    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da1c2756-594d-4018-a886-260b76006523-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.308810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.308635    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da1c2756-594d-4018-a886-260b76006523-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:44:31.583645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.583555    4478 generic.go:356] "Generic (PLEG): container finished" podID="da1c2756-594d-4018-a886-260b76006523" containerID="1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211" exitCode=0
Oct 31 00:44:31.583645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.583608    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"
Oct 31 00:44:31.583824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.583651    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" event={"ID":"da1c2756-594d-4018-a886-260b76006523","Type":"ContainerDied","Data":"1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211"}
Oct 31 00:44:31.583824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.583708    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp" event={"ID":"da1c2756-594d-4018-a886-260b76006523","Type":"ContainerDied","Data":"47ef7f5da91dca9ecb86f8dd61f98a35ebc48417f17d005a2cfa0241970c7040"}
Oct 31 00:44:31.583824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.583731    4478 scope.go:122] "RemoveContainer" containerID="1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211"
Oct 31 00:44:31.593602 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.593587    4478 scope.go:122] "RemoveContainer" containerID="1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211"
Oct 31 00:44:31.593892 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:44:31.593875    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211\": container with ID starting with 1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211 not found: ID does not exist" containerID="1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211"
Oct 31 00:44:31.593960 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.593901    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211"} err="failed to get container status \"1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211\": rpc error: code = NotFound desc = could not find container \"1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211\": container with ID starting with 1dedc67298653bf82e7ffeceb5a4506312b34aa89af80be6854ceae0eb5b6211 not found: ID does not exist"
Oct 31 00:44:31.748651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.748616    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"]
Oct 31 00:44:31.797739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:31.797713    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-10a7aa-openshift-default-8d4bff587-tkrwp"]
Oct 31 00:44:33.116799 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:33.116768    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da1c2756-594d-4018-a886-260b76006523" path="/var/lib/kubelet/pods/da1c2756-594d-4018-a886-260b76006523/volumes"
Oct 31 00:44:44.776140 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:44:44.776098    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:44:59.716287 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.716248    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"]
Oct 31 00:44:59.716759 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.716745    4478 cpu_manager.go:414] "containerMap: removing container" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy"
Oct 31 00:44:59.716807 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.716759    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy"
Oct 31 00:44:59.716869 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.716859    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="da1c2756-594d-4018-a886-260b76006523" containerName="istio-proxy"
Oct 31 00:44:59.721108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.721086    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:44:59.773648 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.773617    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-ef8827\""
Oct 31 00:44:59.773880 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.773752    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-ef8827-dockercfg-pjd86\""
Oct 31 00:44:59.783682 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.783654    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"]
Oct 31 00:44:59.832402 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.814706    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7blwb\" (UniqueName: \"kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb\") pod \"payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:44:59.856327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.856292    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:44:59.957806 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.957763    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:44:59.958990 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.958899    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:44:59.959292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:44:59.959269    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7blwb\" (UniqueName: \"kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb\") pod \"payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:45:00.038255 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.038170    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7blwb\" (UniqueName: \"kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb\") pod \"payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:45:00.038469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.038450    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:45:00.444262 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.444237    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"]
Oct 31 00:45:00.447067 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:45:00.447037    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb517a62_4860_49aa_9c13_966b3c5ba047.slice/crio-f588afc0b5436ef5ebbcd75a66223bc82c054f2c8ae576cd74b8211e3991142f WatchSource:0}: Error finding container f588afc0b5436ef5ebbcd75a66223bc82c054f2c8ae576cd74b8211e3991142f: Status 404 returned error can't find the container with id f588afc0b5436ef5ebbcd75a66223bc82c054f2c8ae576cd74b8211e3991142f
Oct 31 00:45:00.742106 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.742019    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw" event={"ID":"bb517a62-4860-49aa-9c13-966b3c5ba047","Type":"ContainerStarted","Data":"0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0"}
Oct 31 00:45:00.742106 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.742070    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw" event={"ID":"bb517a62-4860-49aa-9c13-966b3c5ba047","Type":"ContainerStarted","Data":"f588afc0b5436ef5ebbcd75a66223bc82c054f2c8ae576cd74b8211e3991142f"}
Oct 31 00:45:00.742560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.742209    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:45:00.857288 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:00.857223    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw" podStartSLOduration=1.857208216 podStartE2EDuration="1.857208216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:00.85580889 +0000 UTC m=+2533.872473614" watchObservedRunningTime="2026-10-31 00:45:00.857208216 +0000 UTC m=+2533.873872927"
Oct 31 00:45:08.470784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:08.470742    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"]
Oct 31 00:45:08.788671 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:08.788573    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" containerID="cri-o://f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8" gracePeriod=30
Oct 31 00:45:11.750575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:11.750544    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:45:12.489741 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:12.489710    4478 patch_prober.go:29] interesting pod/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:45:12.489914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:12.489767    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:45:14.051302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.051250    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:45:14.130309 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130283    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130324    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130415    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130455    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130547    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130571    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:14.130684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130582    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130620    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130654    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130946 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130720    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo\") pod \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\" (UID: \"d517b74d-a9ea-49f0-9ec2-73358fc5da93\") "
Oct 31 00:45:14.130946 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130892    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:14.131057 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130954    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:14.131057 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.130964    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:14.131168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.131126    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:14.131492 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.131470    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.131569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.131500    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.131569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.131514    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.131569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.131529    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.131569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.131545    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.137913 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.133581    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "kube-api-access-vwp5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:14.137913 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.135429    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:14.138776 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.138302    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:14.141951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.140918    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo" pod "d517b74d-a9ea-49f0-9ec2-73358fc5da93" (UID: "d517b74d-a9ea-49f0-9ec2-73358fc5da93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:45:14.232415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.232375    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.232415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.232417    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwp5h\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-kube-api-access-vwp5h\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.232575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.232427    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.232575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.232436    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d517b74d-a9ea-49f0-9ec2-73358fc5da93-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:14.759424 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:45:14.759365    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:45:14.824784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.824749    4478 generic.go:356] "Generic (PLEG): container finished" podID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerID="f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8" exitCode=0
Oct 31 00:45:14.824981 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.824843    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"
Oct 31 00:45:14.824981 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.824838    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" event={"ID":"d517b74d-a9ea-49f0-9ec2-73358fc5da93","Type":"ContainerDied","Data":"f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8"}
Oct 31 00:45:14.825109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.825004    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb" event={"ID":"d517b74d-a9ea-49f0-9ec2-73358fc5da93","Type":"ContainerDied","Data":"99e0f2b98f01bf2319b57127c7eb917e8391c7064fad38c249c95d5efcde3f90"}
Oct 31 00:45:14.825109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.825041    4478 scope.go:122] "RemoveContainer" containerID="f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8"
Oct 31 00:45:14.835417 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.835373    4478 scope.go:122] "RemoveContainer" containerID="4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"
Oct 31 00:45:14.843773 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.843757    4478 scope.go:122] "RemoveContainer" containerID="f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8"
Oct 31 00:45:14.844023 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:45:14.844006    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8\": container with ID starting with f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8 not found: ID does not exist" containerID="f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8"
Oct 31 00:45:14.844070 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.844031    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8"} err="failed to get container status \"f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8\": rpc error: code = NotFound desc = could not find container \"f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8\": container with ID starting with f6e42cbad4ba5c22f92ff0787aabea5360b21b4a957dd6cf6994663dadc6cdc8 not found: ID does not exist"
Oct 31 00:45:14.844070 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.844049    4478 scope.go:122] "RemoveContainer" containerID="4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"
Oct 31 00:45:14.844257 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:45:14.844243    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206\": container with ID starting with 4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206 not found: ID does not exist" containerID="4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"
Oct 31 00:45:14.844304 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.844261    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206"} err="failed to get container status \"4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206\": rpc error: code = NotFound desc = could not find container \"4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206\": container with ID starting with 4e03b01b817857b6725aec36aaa71e3de3773cd2666d5ddaafd6bdd4f9ef9206 not found: ID does not exist"
Oct 31 00:45:14.989119 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:14.989090    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"]
Oct 31 00:45:15.038449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:15.038347    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-726d5b-openshift-default-78ddc5479-nqmmb"]
Oct 31 00:45:15.117717 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:15.117675    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" path="/var/lib/kubelet/pods/d517b74d-a9ea-49f0-9ec2-73358fc5da93/volumes"
Oct 31 00:45:27.117842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.117810    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"]
Oct 31 00:45:27.118643 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.118621    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy"
Oct 31 00:45:27.118754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.118644    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy"
Oct 31 00:45:27.118754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.118683    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy"
Oct 31 00:45:27.118754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.118690    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy"
Oct 31 00:45:27.118868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.118816    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy"
Oct 31 00:45:27.118868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.118831    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d517b74d-a9ea-49f0-9ec2-73358fc5da93" containerName="istio-proxy"
Oct 31 00:45:27.123088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.123070    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.175582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.175551    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-31144e-openshift-default-dockercfg-qt86z\""
Oct 31 00:45:27.227592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.227559    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"]
Oct 31 00:45:27.296243 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.296201    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.297449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.297413    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.298728 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.298679    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7g77\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.305049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.302469    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.305368 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.305329    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.305634 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.305615    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.305788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.305770    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.305986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.305966    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.306145 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.306130    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407408    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407476    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407515    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407659    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407774 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407701    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407774 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407735    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7g77\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407771    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407809    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407766    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.407887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.407877    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.408097 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.408020    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.408162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.408099    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.408470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.408443    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.408754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.408681    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.410595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.410574    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.410693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.410592    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.470829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.470789    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.471032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.471011    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7g77\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77\") pod \"e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") " pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:27.736491 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:27.736456    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:28.171105 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:28.171079    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"]
Oct 31 00:45:28.176513 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:45:28.176482    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda695baf2_603d_4156_9404_3e92e926cf22.slice/crio-421ad87b3247bfc3b017f8d12ab47c2f055423d732e1b9c685c548b37985e9bd WatchSource:0}: Error finding container 421ad87b3247bfc3b017f8d12ab47c2f055423d732e1b9c685c548b37985e9bd: Status 404 returned error can't find the container with id 421ad87b3247bfc3b017f8d12ab47c2f055423d732e1b9c685c548b37985e9bd
Oct 31 00:45:28.178592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:28.178550    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:28.178684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:28.178641    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:28.178728 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:28.178683    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:28.919848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:28.919807    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" event={"ID":"a695baf2-603d-4156-9404-3e92e926cf22","Type":"ContainerStarted","Data":"0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"}
Oct 31 00:45:28.920022 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:28.919879    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" event={"ID":"a695baf2-603d-4156-9404-3e92e926cf22","Type":"ContainerStarted","Data":"421ad87b3247bfc3b017f8d12ab47c2f055423d732e1b9c685c548b37985e9bd"}
Oct 31 00:45:29.090620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:29.090570    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podStartSLOduration=2.09055724 podStartE2EDuration="2.09055724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:29.089275716 +0000 UTC m=+2562.105940446" watchObservedRunningTime="2026-10-31 00:45:29.09055724 +0000 UTC m=+2562.107221952"
Oct 31 00:45:29.737202 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:29.737168    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:29.744689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:29.744663    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:29.925136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:29.925107    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:29.926001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:29.925980    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:45:37.764957 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.764922    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"]
Oct 31 00:45:37.770079 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.770062    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:37.820512 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.820485    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:37.820651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.820605    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6kxv\" (UniqueName: \"kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv\") pod \"payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:37.826130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.826099    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"]
Oct 31 00:45:37.872991 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.872916    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-31144e\""
Oct 31 00:45:37.873594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.873292    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-31144e-dockercfg-8bckm\""
Oct 31 00:45:37.923251 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.923202    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:37.924544 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.924514    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6kxv\" (UniqueName: \"kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv\") pod \"payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:37.925142 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.924331    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:37.987418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:37.986277    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6kxv\" (UniqueName: \"kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv\") pod \"payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:38.085107 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:38.085038    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:38.473735 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:38.473707    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"]
Oct 31 00:45:38.476772 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:45:38.476742    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9dc7df_da5f_462f_99ab_2a13ed2a1cfb.slice/crio-c8a3be257608ad9d698354d5d12f2ddfb7a28afdfb1d50c386272bc36bf02dad WatchSource:0}: Error finding container c8a3be257608ad9d698354d5d12f2ddfb7a28afdfb1d50c386272bc36bf02dad: Status 404 returned error can't find the container with id c8a3be257608ad9d698354d5d12f2ddfb7a28afdfb1d50c386272bc36bf02dad
Oct 31 00:45:38.972696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:38.972643    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk" event={"ID":"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb","Type":"ContainerStarted","Data":"56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5"}
Oct 31 00:45:38.973103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:38.972725    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk" event={"ID":"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb","Type":"ContainerStarted","Data":"c8a3be257608ad9d698354d5d12f2ddfb7a28afdfb1d50c386272bc36bf02dad"}
Oct 31 00:45:38.973103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:38.972757    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:39.089478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:39.089414    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk" podStartSLOduration=2.089372442 podStartE2EDuration="2.089372442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:39.087656938 +0000 UTC m=+2572.104321705" watchObservedRunningTime="2026-10-31 00:45:39.089372442 +0000 UTC m=+2572.106037155"
Oct 31 00:45:39.893449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:39.893418    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"]
Oct 31 00:45:39.893763 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:39.893611    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy" containerID="cri-o://3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92" gracePeriod=30
Oct 31 00:45:44.784036 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:45:44.784000    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:45:45.159631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.159609    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:45:45.202727 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.202704    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.202890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.202746    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.202890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.202785    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.202890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.202816    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.202890 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.202856    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.203101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.202893    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.203101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203083    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.203208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203127    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.203208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203175    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc\") pod \"e397d119-10c6-4fef-960b-aea1b23923cd\" (UID: \"e397d119-10c6-4fef-960b-aea1b23923cd\") "
Oct 31 00:45:45.203485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203170    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:45.203485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203310    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:45.203485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203325    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:45.203693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203452    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:45.203748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203616    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:45:45.203748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203742    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.203851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203764    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.203851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203778    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.203851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.203792    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.205497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.205457    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:45:45.205600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.205578    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "kube-api-access-2pbxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:45.205731 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.205709    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:45:45.206075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.206051    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo" pod "e397d119-10c6-4fef-960b-aea1b23923cd" (UID: "e397d119-10c6-4fef-960b-aea1b23923cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:45:45.310404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.305555    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e397d119-10c6-4fef-960b-aea1b23923cd-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.310404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.305600    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pbxc\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-kube-api-access-2pbxc\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.310404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.305617    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e397d119-10c6-4fef-960b-aea1b23923cd-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.310404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.305631    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e397d119-10c6-4fef-960b-aea1b23923cd-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:45.310404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:45.305646    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e397d119-10c6-4fef-960b-aea1b23923cd-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:45:46.012471 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.012430    4478 generic.go:356] "Generic (PLEG): container finished" podID="e397d119-10c6-4fef-960b-aea1b23923cd" containerID="3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92" exitCode=0
Oct 31 00:45:46.012924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.012476    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"
Oct 31 00:45:46.012924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.012507    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" event={"ID":"e397d119-10c6-4fef-960b-aea1b23923cd","Type":"ContainerDied","Data":"3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92"}
Oct 31 00:45:46.012924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.012579    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7" event={"ID":"e397d119-10c6-4fef-960b-aea1b23923cd","Type":"ContainerDied","Data":"6330f8da4019010d398dab1085e1e6e01260715a2182e6a036186060b0b6034d"}
Oct 31 00:45:46.012924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.012610    4478 scope.go:122] "RemoveContainer" containerID="3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92"
Oct 31 00:45:46.032667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.032640    4478 scope.go:122] "RemoveContainer" containerID="3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92"
Oct 31 00:45:46.032982 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:45:46.032962    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92\": container with ID starting with 3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92 not found: ID does not exist" containerID="3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92"
Oct 31 00:45:46.033064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.032991    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92"} err="failed to get container status \"3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92\": rpc error: code = NotFound desc = could not find container \"3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92\": container with ID starting with 3696f281cb11ddc10f5f1247726a3382d087da6e19d9c519a3735c07e209cf92 not found: ID does not exist"
Oct 31 00:45:46.177076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.177045    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"]
Oct 31 00:45:46.181305 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:46.181280    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-bfpd7"]
Oct 31 00:45:47.117583 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:47.117551    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" path="/var/lib/kubelet/pods/e397d119-10c6-4fef-960b-aea1b23923cd/volumes"
Oct 31 00:45:49.330132 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.330100    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"]
Oct 31 00:45:49.330868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.330845    4478 cpu_manager.go:414] "containerMap: removing container" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy"
Oct 31 00:45:49.330868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.330870    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy"
Oct 31 00:45:49.331017 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.330988    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="e397d119-10c6-4fef-960b-aea1b23923cd" containerName="istio-proxy"
Oct 31 00:45:49.335670 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.335652    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.390406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.389457    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-bf38e4-openshift-default-dockercfg-hfbvm\""
Oct 31 00:45:49.450953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.450919    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"]
Oct 31 00:45:49.468997 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.468962    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469122    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469301    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469376    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469429    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gntb6\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469517    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469579    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469633    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.471134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.469724    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.570483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570447    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.570694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570487    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.570694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570510    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.570694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570558    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.570694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570632    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.570694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570675    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570735    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570772    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570798    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gntb6\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.570956    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571227 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.571190    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.571597    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571726 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.571615    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.571781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.571719    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.573585 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.573562    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.573821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.573802    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.631320 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.631248    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.631527 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.631494    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gntb6\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6\") pod \"e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") " pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.648845 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.648819    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:49.982063 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:49.982037    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:45:50.092622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:50.092595    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"]
Oct 31 00:45:50.094579 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:45:50.094551    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4a155e_335a_4431_9d0c_1e57d02daf1a.slice/crio-f4306cb2caf3a16c9f9fbec1a78d1ceb9562298acb928e9cbee5f6c2e7ef5569 WatchSource:0}: Error finding container f4306cb2caf3a16c9f9fbec1a78d1ceb9562298acb928e9cbee5f6c2e7ef5569: Status 404 returned error can't find the container with id f4306cb2caf3a16c9f9fbec1a78d1ceb9562298acb928e9cbee5f6c2e7ef5569
Oct 31 00:45:50.097165 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:50.097127    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:50.097261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:50.097203    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:50.097261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:50.097249    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:51.040459 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:51.040419    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" event={"ID":"3b4a155e-335a-4431-9d0c-1e57d02daf1a","Type":"ContainerStarted","Data":"2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"}
Oct 31 00:45:51.040959 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:51.040492    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" event={"ID":"3b4a155e-335a-4431-9d0c-1e57d02daf1a","Type":"ContainerStarted","Data":"f4306cb2caf3a16c9f9fbec1a78d1ceb9562298acb928e9cbee5f6c2e7ef5569"}
Oct 31 00:45:51.162401 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:51.162326    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podStartSLOduration=2.162309559 podStartE2EDuration="2.162309559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:51.160418886 +0000 UTC m=+2584.177083628" watchObservedRunningTime="2026-10-31 00:45:51.162309559 +0000 UTC m=+2584.178974272"
Oct 31 00:45:51.649217 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:51.649181    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:51.655332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:51.655310    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:52.045352 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:52.045312    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:52.046526 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:52.046497    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:45:55.026399 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.026352    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"]
Oct 31 00:45:55.030064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.030044    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.133186 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.133147    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"]
Oct 31 00:45:55.135052 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.135031    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e1ca71-openshift-default-dockercfg-sh7lb\""
Oct 31 00:45:55.139689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.139664    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.139822 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.139743    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9bdg\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.139883 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.139837    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.139922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.139882    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.139972 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.139923    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.140053 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.140023    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.140129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.140110    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.140188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.140146    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.140188 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.140178    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.241706 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241670    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.241872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241765    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9bdg\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.241872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241792    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.241872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241822    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241870    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241927    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.241981    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.242009    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.242036    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.242174    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.242249    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.242672    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.242802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.242765    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.243291 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.243259    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.244545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.244525    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.244764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.244744    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.302448 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.302358    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.302549 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.302515    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9bdg\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.344856 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.344831    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:55.788060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.788031    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"]
Oct 31 00:45:55.791422 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:45:55.791362    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86ad1b60_b6f2_485b_bcc1_a95f347faa6e.slice/crio-326cdcf0f0e9861671cfd902f04ad87ea0ac57641270e4b12fcf96e79ff944c1 WatchSource:0}: Error finding container 326cdcf0f0e9861671cfd902f04ad87ea0ac57641270e4b12fcf96e79ff944c1: Status 404 returned error can't find the container with id 326cdcf0f0e9861671cfd902f04ad87ea0ac57641270e4b12fcf96e79ff944c1
Oct 31 00:45:55.794594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.794560    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:55.794700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.794637    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:55.794700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:55.794684    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:45:56.071322 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:56.071219    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" event={"ID":"86ad1b60-b6f2-485b-bcc1-a95f347faa6e","Type":"ContainerStarted","Data":"5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055"}
Oct 31 00:45:56.071857 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:56.071330    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" event={"ID":"86ad1b60-b6f2-485b-bcc1-a95f347faa6e","Type":"ContainerStarted","Data":"326cdcf0f0e9861671cfd902f04ad87ea0ac57641270e4b12fcf96e79ff944c1"}
Oct 31 00:45:56.192077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:56.192025    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podStartSLOduration=2.192010241 podStartE2EDuration="2.192010241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:45:56.190442085 +0000 UTC m=+2589.207106823" watchObservedRunningTime="2026-10-31 00:45:56.192010241 +0000 UTC m=+2589.208674977"
Oct 31 00:45:56.352864 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:56.352766    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:45:56.357870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:56.357840    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:45:56.358151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:56.358111    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:45:57.347407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:57.345771    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:45:57.347407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:57.345833    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:45:58.362555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.362511    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:45:58.363088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.362574    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:45:58.526327 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.522141    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"]
Oct 31 00:45:58.527444 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.527417    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.585187 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.585138    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"]
Oct 31 00:45:58.620148 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.619915    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbt6r\" (UniqueName: \"kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r\") pod \"payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.620148 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.620039    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.627665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.627642    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-bf38e4\""
Oct 31 00:45:58.627781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.627659    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-bf38e4-dockercfg-7g6jc\""
Oct 31 00:45:58.721503 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.721464    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.721696 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.721514    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbt6r\" (UniqueName: \"kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r\") pod \"payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.722183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.722162    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.782212 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.782179    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbt6r\" (UniqueName: \"kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r\") pod \"payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:58.892303 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:58.892224    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:45:59.306185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:59.306155    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"]
Oct 31 00:45:59.308197 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:45:59.308163    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00f4823e_4db5_455a_90a6_2bc045f5257f.slice/crio-118bc5f66e68e949841151896949926fbbad1bfc70187f78435be192cd998a9e WatchSource:0}: Error finding container 118bc5f66e68e949841151896949926fbbad1bfc70187f78435be192cd998a9e: Status 404 returned error can't find the container with id 118bc5f66e68e949841151896949926fbbad1bfc70187f78435be192cd998a9e
Oct 31 00:45:59.366023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:59.365611    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:45:59.366023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:45:59.365681    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:46:00.208028 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:00.207991    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t" event={"ID":"00f4823e-4db5-455a-90a6-2bc045f5257f","Type":"ContainerStarted","Data":"3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77"}
Oct 31 00:46:00.208269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:00.208048    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t" event={"ID":"00f4823e-4db5-455a-90a6-2bc045f5257f","Type":"ContainerStarted","Data":"118bc5f66e68e949841151896949926fbbad1bfc70187f78435be192cd998a9e"}
Oct 31 00:46:00.208269 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:00.208261    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:46:00.330908 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:00.330861    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t" podStartSLOduration=2.330839503 podStartE2EDuration="2.330839503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:45:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:00.328643568 +0000 UTC m=+2593.345308332" watchObservedRunningTime="2026-10-31 00:46:00.330839503 +0000 UTC m=+2593.347504242"
Oct 31 00:46:00.349739 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:00.349704    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:46:00.349988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:00.349962    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:46:01.346405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:01.345732    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:46:01.346405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:01.345796    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:46:02.345482 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:02.345441    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused" start-of-body=
Oct 31 00:46:02.345662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:02.345510    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.125:15021/healthz/ready\": dial tcp 10.132.0.125:15021: connect: connection refused"
Oct 31 00:46:03.350151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:03.350116    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:46:03.350571 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:03.350481    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:46:03.351113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:03.351094    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 00:46:11.216842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:11.216814    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:46:14.757061 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:46:14.757015    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:46:21.605550 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:21.605498    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"]
Oct 31 00:46:26.606665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:26.606603    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerName="payload-pre-processing" containerID="cri-o://662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf" gracePeriod=25
Oct 31 00:46:29.013727 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:29.013696    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:46:39.014049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:39.014014    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 00:46:44.756701 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:46:44.756666    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:46:46.875686 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:46.875623    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:46:46.993406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:46.993355    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh\" (UniqueName: \"kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh\") pod \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") "
Oct 31 00:46:46.993575 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:46.993479    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume\") pod \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\" (UID: \"56ce0b99-c09a-475a-beb4-c4ca8f45e130\") "
Oct 31 00:46:46.994358 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:46.994328    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume" pod "56ce0b99-c09a-475a-beb4-c4ca8f45e130" (UID: "56ce0b99-c09a-475a-beb4-c4ca8f45e130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:46:46.995900 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:46.995857    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh" pod "56ce0b99-c09a-475a-beb4-c4ca8f45e130" (UID: "56ce0b99-c09a-475a-beb4-c4ca8f45e130"). InnerVolumeSpecName "kube-api-access-z68gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:46:47.095116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.095083    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56ce0b99-c09a-475a-beb4-c4ca8f45e130-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:46:47.095116 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.095110    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z68gh\" (UniqueName: \"kubernetes.io/projected/56ce0b99-c09a-475a-beb4-c4ca8f45e130-kube-api-access-z68gh\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:46:47.496313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.496276    4478 generic.go:356] "Generic (PLEG): container finished" podID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerID="662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf" exitCode=0
Oct 31 00:46:47.496500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.496324    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" event={"ID":"56ce0b99-c09a-475a-beb4-c4ca8f45e130","Type":"ContainerDied","Data":"662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf"}
Oct 31 00:46:47.496500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.496342    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"
Oct 31 00:46:47.496500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.496435    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs" event={"ID":"56ce0b99-c09a-475a-beb4-c4ca8f45e130","Type":"ContainerDied","Data":"ab38bbf3194d929dd7a3887c790bb11c63274d5b6a71393798699412222496d1"}
Oct 31 00:46:47.496500 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.496468    4478 scope.go:122] "RemoveContainer" containerID="662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf"
Oct 31 00:46:47.505497 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.505472    4478 scope.go:122] "RemoveContainer" containerID="662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf"
Oct 31 00:46:47.505802 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:46:47.505783    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf\": container with ID starting with 662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf not found: ID does not exist" containerID="662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf"
Oct 31 00:46:47.505851 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.505811    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf"} err="failed to get container status \"662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf\": rpc error: code = NotFound desc = could not find container \"662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf\": container with ID starting with 662ad5b10d27e84b4c6f89e0aab7f62a6047a906aa1342cf564a209ebab63cdf not found: ID does not exist"
Oct 31 00:46:47.659685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.659655    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"]
Oct 31 00:46:47.710665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:47.710602    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-87b0f8-57f5446959-7d8rs"]
Oct 31 00:46:49.119261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:49.119223    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" path="/var/lib/kubelet/pods/56ce0b99-c09a-475a-beb4-c4ca8f45e130/volumes"
Oct 31 00:46:59.096164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.096083    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"]
Oct 31 00:46:59.096692 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.096678    4478 cpu_manager.go:414] "containerMap: removing container" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerName="payload-pre-processing"
Oct 31 00:46:59.096738 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.096695    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerName="payload-pre-processing"
Oct 31 00:46:59.096784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.096774    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="56ce0b99-c09a-475a-beb4-c4ca8f45e130" containerName="payload-pre-processing"
Oct 31 00:46:59.101511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.101491    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.153504 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.153474    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-bf38e4-serving-cert\""
Oct 31 00:46:59.205533 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.205503    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"]
Oct 31 00:46:59.233065 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.233035    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.233248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.233079    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vhtw\" (UniqueName: \"kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.233248 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.233181    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.233374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.233354    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.334565 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.334531    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.334770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.334574    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vhtw\" (UniqueName: \"kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.334770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.334635    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.334770 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:46:59.334694    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-serving-cert: secret "maas-api-e2e-praxis-b-bf38e4-serving-cert" not found
Oct 31 00:46:59.334770 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.334706    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.334770 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:46:59.334773    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls podName:fa263a05-e0f4-47bc-a0e0-050dcaa06ff0 nodeName:}" failed. No retries permitted until 2026-10-31 00:46:59.834753484 +0000 UTC m=+2652.851418179 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls") pod "maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" (UID: "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0") : secret "maas-api-e2e-praxis-b-bf38e4-serving-cert" not found
Oct 31 00:46:59.335568 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.335546    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.355311 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.354769    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.395904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.395873    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vhtw\" (UniqueName: \"kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.841164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.841114    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:46:59.843815 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:46:59.843781    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\") pod \"maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:47:00.017277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:00.017246    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:47:00.400239 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:00.400211    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"]
Oct 31 00:47:00.402453 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:47:00.402422    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa263a05_e0f4_47bc_a0e0_050dcaa06ff0.slice/crio-e9f69b7104f9e0806ef641d62994324671f789e9eab9948ad01682b0ff608cef WatchSource:0}: Error finding container e9f69b7104f9e0806ef641d62994324671f789e9eab9948ad01682b0ff608cef: Status 404 returned error can't find the container with id e9f69b7104f9e0806ef641d62994324671f789e9eab9948ad01682b0ff608cef
Oct 31 00:47:00.575584 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:00.575542    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" event={"ID":"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0","Type":"ContainerStarted","Data":"e9f69b7104f9e0806ef641d62994324671f789e9eab9948ad01682b0ff608cef"}
Oct 31 00:47:01.582597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:01.582506    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" event={"ID":"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0","Type":"ContainerStarted","Data":"e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206"}
Oct 31 00:47:01.582952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:01.582600    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:47:01.700582 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:01.700528    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" podStartSLOduration=1.848964695 podStartE2EDuration="2.700514206s" totalImagesPullingTime="851.549511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:47:01.699711052 +0000 UTC m=+2654.716375764" watchObservedRunningTime="2026-10-31 00:47:01.700514206 +0000 UTC m=+2654.717178918"
Oct 31 00:47:07.613109 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:07.613082    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:47:11.691096 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:11.691062    4478 generic.go:356] "Generic (PLEG): container finished" podID="a695baf2-603d-4156-9404-3e92e926cf22" containerID="0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519" exitCode=0
Oct 31 00:47:11.691552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:11.691135    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" event={"ID":"a695baf2-603d-4156-9404-3e92e926cf22","Type":"ContainerDied","Data":"0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"}
Oct 31 00:47:11.691552 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:11.691522    4478 scope.go:122] "RemoveContainer" containerID="0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"
Oct 31 00:47:11.693064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:11.693038    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:47:11.693133 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:11.693118    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:47:11.693173 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:11.693152    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:47:12.698046 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:12.698013    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" event={"ID":"a695baf2-603d-4156-9404-3e92e926cf22","Type":"ContainerStarted","Data":"6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452"}
Oct 31 00:47:12.747406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:12.745446    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:47:12.759410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:12.758583    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:47:12.759410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:12.758675    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:47:13.744440 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:13.744404    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:47:13.745019 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:13.744987    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:47:14.756408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:14.755478    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:47:14.756408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:14.755545    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:47:14.795321 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:47:14.795270    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:47:14.925431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:14.925400    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:47:15.755407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:15.754775    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:47:15.755407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:15.754894    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:47:16.754404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:16.751570    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:47:16.754404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:16.751684    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:47:17.764258 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:17.764213    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Oct 31 00:47:17.764764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:17.764491    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Oct 31 00:47:18.741607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:18.741578    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:47:18.742849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:18.742827    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:47:31.847496 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:31.847460    4478 generic.go:356] "Generic (PLEG): container finished" podID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerID="2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366" exitCode=0
Oct 31 00:47:31.847920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:31.847524    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" event={"ID":"3b4a155e-335a-4431-9d0c-1e57d02daf1a","Type":"ContainerDied","Data":"2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"}
Oct 31 00:47:31.847920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:31.847882    4478 scope.go:122] "RemoveContainer" containerID="2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"
Oct 31 00:47:31.849435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:31.849405    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:47:31.849499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:31.849483    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:47:31.849535 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:31.849520    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:47:32.856443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:32.856405    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" event={"ID":"3b4a155e-335a-4431-9d0c-1e57d02daf1a","Type":"ContainerStarted","Data":"bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488"}
Oct 31 00:47:33.651089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:33.651058    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:47:33.651494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:33.651467    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:47:33.651700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:33.651671    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:47:34.652414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:34.650790    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:47:34.652414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:34.650853    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:47:35.650407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:35.649764    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:47:35.650407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:35.649823    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:47:36.653407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:36.652765    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:47:36.653407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:36.652831    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:47:37.047408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:37.046479    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:47:37.652786 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:37.652749    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body=
Oct 31 00:47:37.653083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:37.653054    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused"
Oct 31 00:47:38.664753 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:38.664722    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:47:38.668653 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:47:38.668627    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:47:44.758080 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:47:44.758044    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:48:08.221787 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:08.221751    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"]
Oct 31 00:48:13.222304 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.222261    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw" podUID="bb517a62-4860-49aa-9c13-966b3c5ba047" containerName="payload-pre-processing" containerID="cri-o://0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0" gracePeriod=25
Oct 31 00:48:13.472812 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.472760    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:48:13.536701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.536672    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb\" (UniqueName: \"kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb\") pod \"bb517a62-4860-49aa-9c13-966b3c5ba047\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") "
Oct 31 00:48:13.536701 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.536705    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume\") pod \"bb517a62-4860-49aa-9c13-966b3c5ba047\" (UID: \"bb517a62-4860-49aa-9c13-966b3c5ba047\") "
Oct 31 00:48:13.537113 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.537086    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume" pod "bb517a62-4860-49aa-9c13-966b3c5ba047" (UID: "bb517a62-4860-49aa-9c13-966b3c5ba047"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:13.537196 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.537178    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb517a62-4860-49aa-9c13-966b3c5ba047-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:13.539126 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.539101    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb" pod "bb517a62-4860-49aa-9c13-966b3c5ba047" (UID: "bb517a62-4860-49aa-9c13-966b3c5ba047"). InnerVolumeSpecName "kube-api-access-7blwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:13.638914 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:13.638883    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7blwb\" (UniqueName: \"kubernetes.io/projected/bb517a62-4860-49aa-9c13-966b3c5ba047-kube-api-access-7blwb\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:14.117784 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.117747    4478 generic.go:356] "Generic (PLEG): container finished" podID="bb517a62-4860-49aa-9c13-966b3c5ba047" containerID="0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0" exitCode=0
Oct 31 00:48:14.117955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.117798    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"
Oct 31 00:48:14.117955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.117797    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw" event={"ID":"bb517a62-4860-49aa-9c13-966b3c5ba047","Type":"ContainerDied","Data":"0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0"}
Oct 31 00:48:14.117955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.117920    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw" event={"ID":"bb517a62-4860-49aa-9c13-966b3c5ba047","Type":"ContainerDied","Data":"f588afc0b5436ef5ebbcd75a66223bc82c054f2c8ae576cd74b8211e3991142f"}
Oct 31 00:48:14.117955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.117948    4478 scope.go:122] "RemoveContainer" containerID="0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0"
Oct 31 00:48:14.128005 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.127989    4478 scope.go:122] "RemoveContainer" containerID="0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0"
Oct 31 00:48:14.128269 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:14.128247    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0\": container with ID starting with 0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0 not found: ID does not exist" containerID="0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0"
Oct 31 00:48:14.128348 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.128276    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0"} err="failed to get container status \"0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0\": rpc error: code = NotFound desc = could not find container \"0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0\": container with ID starting with 0cf81224d24cfad30f9c6f9d33f6ed75092b7db36cca2e53b3e3496183e022e0 not found: ID does not exist"
Oct 31 00:48:14.283000 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.282968    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"]
Oct 31 00:48:14.332184 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:14.332147    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-ef8827-7fd8bd4ddc-bqskw"]
Oct 31 00:48:14.757585 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:14.757549    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:48:15.117485 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:15.117410    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb517a62-4860-49aa-9c13-966b3c5ba047" path="/var/lib/kubelet/pods/bb517a62-4860-49aa-9c13-966b3c5ba047/volumes"
Oct 31 00:48:19.770510 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:19.770481    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"]
Oct 31 00:48:19.875611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:19.875576    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"]
Oct 31 00:48:20.458059 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.458017    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"]
Oct 31 00:48:20.458276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.458199    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" podUID="4ccf352b-35e8-4929-aa04-7cff9e08d928" containerName="maas-api" containerID="cri-o://37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852" gracePeriod=30
Oct 31 00:48:20.756484 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.755944    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:48:20.818961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.818929    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\") pod \"4ccf352b-35e8-4929-aa04-7cff9e08d928\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") "
Oct 31 00:48:20.819366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.819131    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz\" (UniqueName: \"kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz\") pod \"4ccf352b-35e8-4929-aa04-7cff9e08d928\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") "
Oct 31 00:48:20.819366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.819176    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs\") pod \"4ccf352b-35e8-4929-aa04-7cff9e08d928\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") "
Oct 31 00:48:20.819366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.819273    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca\") pod \"4ccf352b-35e8-4929-aa04-7cff9e08d928\" (UID: \"4ccf352b-35e8-4929-aa04-7cff9e08d928\") "
Oct 31 00:48:20.820210 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.820167    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca" pod "4ccf352b-35e8-4929-aa04-7cff9e08d928" (UID: "4ccf352b-35e8-4929-aa04-7cff9e08d928"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:20.821340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.821315    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls" pod "4ccf352b-35e8-4929-aa04-7cff9e08d928" (UID: "4ccf352b-35e8-4929-aa04-7cff9e08d928"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:20.821592 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.821562    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs" pod "4ccf352b-35e8-4929-aa04-7cff9e08d928" (UID: "4ccf352b-35e8-4929-aa04-7cff9e08d928"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:20.821788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.821764    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz" pod "4ccf352b-35e8-4929-aa04-7cff9e08d928" (UID: "4ccf352b-35e8-4929-aa04-7cff9e08d928"). InnerVolumeSpecName "kube-api-access-2npjz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:20.921523 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.921485    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:20.921681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.921527    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2npjz\" (UniqueName: \"kubernetes.io/projected/4ccf352b-35e8-4929-aa04-7cff9e08d928-kube-api-access-2npjz\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:20.921681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.921546    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ccf352b-35e8-4929-aa04-7cff9e08d928-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:20.921681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:20.921561    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ccf352b-35e8-4929-aa04-7cff9e08d928-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:21.157714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.157678    4478 generic.go:356] "Generic (PLEG): container finished" podID="4ccf352b-35e8-4929-aa04-7cff9e08d928" containerID="37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852" exitCode=0
Oct 31 00:48:21.157886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.157733    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" event={"ID":"4ccf352b-35e8-4929-aa04-7cff9e08d928","Type":"ContainerDied","Data":"37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852"}
Oct 31 00:48:21.157886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.157738    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"
Oct 31 00:48:21.157886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.157780    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf" event={"ID":"4ccf352b-35e8-4929-aa04-7cff9e08d928","Type":"ContainerDied","Data":"b9640ba77e13945e4945423b23d2d16910776ca0ea6603f15f7d0ab3515e63a5"}
Oct 31 00:48:21.157886 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.157804    4478 scope.go:122] "RemoveContainer" containerID="37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852"
Oct 31 00:48:21.167970 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.167949    4478 scope.go:122] "RemoveContainer" containerID="37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852"
Oct 31 00:48:21.168289 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:21.168269    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852\": container with ID starting with 37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852 not found: ID does not exist" containerID="37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852"
Oct 31 00:48:21.168344 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.168302    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852"} err="failed to get container status \"37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852\": rpc error: code = NotFound desc = could not find container \"37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852\": container with ID starting with 37906c9d649c62ac436c4ed4783045963272590ccd7ad982e5b2960c16764852 not found: ID does not exist"
Oct 31 00:48:21.333420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.333370    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"]
Oct 31 00:48:21.338941 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.338915    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-87b0f8-78bbb44c8b-nqpcf"]
Oct 31 00:48:21.593284 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.593248    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"]
Oct 31 00:48:21.593560 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.593536    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" podUID="cd67942f-ee48-4d55-aebf-48bd6756f49e" containerName="maas-api" containerID="cri-o://4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc" gracePeriod=30
Oct 31 00:48:21.872259 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.872234    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:48:21.944169 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.944141    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh\" (UniqueName: \"kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh\") pod \"cd67942f-ee48-4d55-aebf-48bd6756f49e\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") "
Oct 31 00:48:21.944306 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.944252    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca\") pod \"cd67942f-ee48-4d55-aebf-48bd6756f49e\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") "
Oct 31 00:48:21.944306 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.944303    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs\") pod \"cd67942f-ee48-4d55-aebf-48bd6756f49e\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") "
Oct 31 00:48:21.944441 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.944332    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\") pod \"cd67942f-ee48-4d55-aebf-48bd6756f49e\" (UID: \"cd67942f-ee48-4d55-aebf-48bd6756f49e\") "
Oct 31 00:48:21.944634 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.944613    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca" pod "cd67942f-ee48-4d55-aebf-48bd6756f49e" (UID: "cd67942f-ee48-4d55-aebf-48bd6756f49e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:21.944727 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.944713    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd67942f-ee48-4d55-aebf-48bd6756f49e-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:21.946508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.946484    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls" pod "cd67942f-ee48-4d55-aebf-48bd6756f49e" (UID: "cd67942f-ee48-4d55-aebf-48bd6756f49e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:21.946606 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.946514    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs" pod "cd67942f-ee48-4d55-aebf-48bd6756f49e" (UID: "cd67942f-ee48-4d55-aebf-48bd6756f49e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:48:21.946661 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:21.946622    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh" pod "cd67942f-ee48-4d55-aebf-48bd6756f49e" (UID: "cd67942f-ee48-4d55-aebf-48bd6756f49e"). InnerVolumeSpecName "kube-api-access-xdskh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:22.046466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.046434    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:22.046466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.046469    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd67942f-ee48-4d55-aebf-48bd6756f49e-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:22.046667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.046480    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdskh\" (UniqueName: \"kubernetes.io/projected/cd67942f-ee48-4d55-aebf-48bd6756f49e-kube-api-access-xdskh\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:22.168128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.168042    4478 generic.go:356] "Generic (PLEG): container finished" podID="cd67942f-ee48-4d55-aebf-48bd6756f49e" containerID="4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc" exitCode=0
Oct 31 00:48:22.168128 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.168112    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"
Oct 31 00:48:22.168355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.168125    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" event={"ID":"cd67942f-ee48-4d55-aebf-48bd6756f49e","Type":"ContainerDied","Data":"4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc"}
Oct 31 00:48:22.168355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.168184    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh" event={"ID":"cd67942f-ee48-4d55-aebf-48bd6756f49e","Type":"ContainerDied","Data":"91a5b100b61b67ee7a17f7f4a13bb4294903580327c12619138a72fb4562789c"}
Oct 31 00:48:22.168355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.168211    4478 scope.go:122] "RemoveContainer" containerID="4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc"
Oct 31 00:48:22.178206 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.178189    4478 scope.go:122] "RemoveContainer" containerID="4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc"
Oct 31 00:48:22.178488 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:22.178467    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc\": container with ID starting with 4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc not found: ID does not exist" containerID="4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc"
Oct 31 00:48:22.178545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.178497    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc"} err="failed to get container status \"4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc\": rpc error: code = NotFound desc = could not find container \"4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc\": container with ID starting with 4435edc57f34d003f950d9e2c2f8b21d20c4dc93213d40c7e3ab249ec32b17cc not found: ID does not exist"
Oct 31 00:48:22.333927 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.333900    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"]
Oct 31 00:48:22.339429 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:22.339403    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-7ae676-7c97bbcddf-d52sh"]
Oct 31 00:48:23.122044 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:23.122014    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccf352b-35e8-4929-aa04-7cff9e08d928" path="/var/lib/kubelet/pods/4ccf352b-35e8-4929-aa04-7cff9e08d928/volumes"
Oct 31 00:48:23.122445 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:23.122411    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd67942f-ee48-4d55-aebf-48bd6756f49e" path="/var/lib/kubelet/pods/cd67942f-ee48-4d55-aebf-48bd6756f49e/volumes"
Oct 31 00:48:24.771759 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:24.771719    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz" podUID="64d3be45-e851-4c25-85fd-bf83a26e2bfa" containerName="payload-processing" containerID="cri-o://933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c" gracePeriod=25
Oct 31 00:48:24.877110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:24.877051    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts" podUID="15031097-609b-4830-a87b-eb2ed9494574" containerName="payload-pre-processing" containerID="cri-o://c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9" gracePeriod=25
Oct 31 00:48:25.062121 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.062040    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:48:25.159800 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.159778    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:48:25.183629 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.183596    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume" pod "64d3be45-e851-4c25-85fd-bf83a26e2bfa" (UID: "64d3be45-e851-4c25-85fd-bf83a26e2bfa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:25.184464 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.184440    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume\") pod \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") "
Oct 31 00:48:25.184664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.184650    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8\" (UniqueName: \"kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8\") pod \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\" (UID: \"64d3be45-e851-4c25-85fd-bf83a26e2bfa\") "
Oct 31 00:48:25.187685 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.187657    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8" pod "64d3be45-e851-4c25-85fd-bf83a26e2bfa" (UID: "64d3be45-e851-4c25-85fd-bf83a26e2bfa"). InnerVolumeSpecName "kube-api-access-nfxq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:25.191136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.191111    4478 generic.go:356] "Generic (PLEG): container finished" podID="64d3be45-e851-4c25-85fd-bf83a26e2bfa" containerID="933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c" exitCode=0
Oct 31 00:48:25.191238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.191153    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"
Oct 31 00:48:25.191238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.191204    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz" event={"ID":"64d3be45-e851-4c25-85fd-bf83a26e2bfa","Type":"ContainerDied","Data":"933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c"}
Oct 31 00:48:25.191336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.191287    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz" event={"ID":"64d3be45-e851-4c25-85fd-bf83a26e2bfa","Type":"ContainerDied","Data":"49f0166d689b5faa3d72d78d43293b80c59c1c99b99ac74570510fc626a31166"}
Oct 31 00:48:25.191336 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.191322    4478 scope.go:122] "RemoveContainer" containerID="933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c"
Oct 31 00:48:25.192863 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.192834    4478 generic.go:356] "Generic (PLEG): container finished" podID="15031097-609b-4830-a87b-eb2ed9494574" containerID="c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9" exitCode=0
Oct 31 00:48:25.192985 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.192970    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"
Oct 31 00:48:25.193136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.192975    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts" event={"ID":"15031097-609b-4830-a87b-eb2ed9494574","Type":"ContainerDied","Data":"c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9"}
Oct 31 00:48:25.193136 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.193089    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts" event={"ID":"15031097-609b-4830-a87b-eb2ed9494574","Type":"ContainerDied","Data":"12cb0c9173d0b6603703137cc87a9114c5cc342c3f5916570ebc24df028fdb25"}
Oct 31 00:48:25.204323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.204304    4478 scope.go:122] "RemoveContainer" containerID="933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c"
Oct 31 00:48:25.204657 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:25.204634    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c\": container with ID starting with 933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c not found: ID does not exist" containerID="933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c"
Oct 31 00:48:25.204704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.204671    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c"} err="failed to get container status \"933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c\": rpc error: code = NotFound desc = could not find container \"933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c\": container with ID starting with 933f1448e509b86ba6f0522896e042d4c918eafe7f6ca57b0865660bd735832c not found: ID does not exist"
Oct 31 00:48:25.204704 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.204696    4478 scope.go:122] "RemoveContainer" containerID="c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9"
Oct 31 00:48:25.213503 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.213478    4478 scope.go:122] "RemoveContainer" containerID="c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9"
Oct 31 00:48:25.213781 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:25.213762    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9\": container with ID starting with c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9 not found: ID does not exist" containerID="c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9"
Oct 31 00:48:25.213844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.213788    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9"} err="failed to get container status \"c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9\": rpc error: code = NotFound desc = could not find container \"c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9\": container with ID starting with c83da7ba49e395f0920e41dddf9258d62d51de14793723115d95a45ff61cf0d9 not found: ID does not exist"
Oct 31 00:48:25.286214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.286160    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv\" (UniqueName: \"kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv\") pod \"15031097-609b-4830-a87b-eb2ed9494574\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") "
Oct 31 00:48:25.286332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.286233    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume\") pod \"15031097-609b-4830-a87b-eb2ed9494574\" (UID: \"15031097-609b-4830-a87b-eb2ed9494574\") "
Oct 31 00:48:25.286611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.286590    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxq8\" (UniqueName: \"kubernetes.io/projected/64d3be45-e851-4c25-85fd-bf83a26e2bfa-kube-api-access-nfxq8\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:25.286677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.286617    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64d3be45-e851-4c25-85fd-bf83a26e2bfa-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:25.286677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.286648    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume" pod "15031097-609b-4830-a87b-eb2ed9494574" (UID: "15031097-609b-4830-a87b-eb2ed9494574"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:25.288310 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.288291    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv" pod "15031097-609b-4830-a87b-eb2ed9494574" (UID: "15031097-609b-4830-a87b-eb2ed9494574"). InnerVolumeSpecName "kube-api-access-hh9pv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:25.387548 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.387519    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh9pv\" (UniqueName: \"kubernetes.io/projected/15031097-609b-4830-a87b-eb2ed9494574-kube-api-access-hh9pv\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:25.387699 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.387554    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/15031097-609b-4830-a87b-eb2ed9494574-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:25.405008 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.404981    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"]
Oct 31 00:48:25.409677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.409655    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-6a693b-65cb466b5c-qcxwz"]
Oct 31 00:48:25.613372 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.613296    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"]
Oct 31 00:48:25.617569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:25.617545    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-6a693b-7db7bc7b47-bg5ts"]
Oct 31 00:48:26.200407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.200328    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"]
Oct 31 00:48:26.200930 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.200911    4478 cpu_manager.go:414] "containerMap: removing container" podUID="cd67942f-ee48-4d55-aebf-48bd6756f49e" containerName="maas-api"
Oct 31 00:48:26.200998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.200933    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd67942f-ee48-4d55-aebf-48bd6756f49e" containerName="maas-api"
Oct 31 00:48:26.200998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.200950    4478 cpu_manager.go:414] "containerMap: removing container" podUID="bb517a62-4860-49aa-9c13-966b3c5ba047" containerName="payload-pre-processing"
Oct 31 00:48:26.200998 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.200957    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb517a62-4860-49aa-9c13-966b3c5ba047" containerName="payload-pre-processing"
Oct 31 00:48:26.201089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201006    4478 cpu_manager.go:414] "containerMap: removing container" podUID="15031097-609b-4830-a87b-eb2ed9494574" containerName="payload-pre-processing"
Oct 31 00:48:26.201089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201012    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15031097-609b-4830-a87b-eb2ed9494574" containerName="payload-pre-processing"
Oct 31 00:48:26.201089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201024    4478 cpu_manager.go:414] "containerMap: removing container" podUID="64d3be45-e851-4c25-85fd-bf83a26e2bfa" containerName="payload-processing"
Oct 31 00:48:26.201089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201029    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64d3be45-e851-4c25-85fd-bf83a26e2bfa" containerName="payload-processing"
Oct 31 00:48:26.201089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201047    4478 cpu_manager.go:414] "containerMap: removing container" podUID="4ccf352b-35e8-4929-aa04-7cff9e08d928" containerName="maas-api"
Oct 31 00:48:26.201089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201052    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccf352b-35e8-4929-aa04-7cff9e08d928" containerName="maas-api"
Oct 31 00:48:26.201264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201127    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="15031097-609b-4830-a87b-eb2ed9494574" containerName="payload-pre-processing"
Oct 31 00:48:26.201264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201135    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="64d3be45-e851-4c25-85fd-bf83a26e2bfa" containerName="payload-processing"
Oct 31 00:48:26.201264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201143    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccf352b-35e8-4929-aa04-7cff9e08d928" containerName="maas-api"
Oct 31 00:48:26.201264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201149    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd67942f-ee48-4d55-aebf-48bd6756f49e" containerName="maas-api"
Oct 31 00:48:26.201264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.201154    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb517a62-4860-49aa-9c13-966b3c5ba047" containerName="payload-pre-processing"
Oct 31 00:48:26.205617 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.205601    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.297722 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.297682    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.297971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.297746    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.297971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.297853    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.297971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.297875    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.297971 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.297965    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.298181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.297984    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.298181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.298012    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x9t5\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.298181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.298143    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.298301 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.298187    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.312606 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.312568    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"]
Oct 31 00:48:26.399788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.399755    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.399788 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.399789    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400017 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.399902    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400017 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.399942    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400017 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.399986    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x9t5\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400059    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400091    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400166    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400167    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400202    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400340 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400295    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400531 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400468    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400537    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.400911 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.400883    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.402557 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.402538    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.402817 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.402801    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.461490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.461416    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.461636 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.461543    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x9t5\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.519091 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.519063    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:26.953181 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.953157    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"]
Oct 31 00:48:26.956414 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:48:26.956351    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bfa2710_f806_49cc_99dc_ae521f1c60ca.slice/crio-0e40910e8d3c6ec162bd928b210c1972eae263efaf36dc9a80550ebfc8582067 WatchSource:0}: Error finding container 0e40910e8d3c6ec162bd928b210c1972eae263efaf36dc9a80550ebfc8582067: Status 404 returned error can't find the container with id 0e40910e8d3c6ec162bd928b210c1972eae263efaf36dc9a80550ebfc8582067
Oct 31 00:48:26.959900 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.959861    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:48:26.960015 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.959947    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:48:26.960015 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:26.959991    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:48:27.117651 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:27.117609    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15031097-609b-4830-a87b-eb2ed9494574" path="/var/lib/kubelet/pods/15031097-609b-4830-a87b-eb2ed9494574/volumes"
Oct 31 00:48:27.118083 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:27.118061    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64d3be45-e851-4c25-85fd-bf83a26e2bfa" path="/var/lib/kubelet/pods/64d3be45-e851-4c25-85fd-bf83a26e2bfa/volumes"
Oct 31 00:48:27.209802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:27.209717    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" event={"ID":"0bfa2710-f806-49cc-99dc-ae521f1c60ca","Type":"ContainerStarted","Data":"62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921"}
Oct 31 00:48:27.209802 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:27.209785    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" event={"ID":"0bfa2710-f806-49cc-99dc-ae521f1c60ca","Type":"ContainerStarted","Data":"0e40910e8d3c6ec162bd928b210c1972eae263efaf36dc9a80550ebfc8582067"}
Oct 31 00:48:27.333706 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:27.333636    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podStartSLOduration=1.333616402 podStartE2EDuration="1.333616402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:27.329325601 +0000 UTC m=+2740.345990331" watchObservedRunningTime="2026-10-31 00:48:27.333616402 +0000 UTC m=+2740.350281117"
Oct 31 00:48:27.519271 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:27.519197    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:28.521943 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:28.521866    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:48:28.521943 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:28.521919    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 00:48:29.520072 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:29.520036    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:48:29.520256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:29.520095    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 00:48:30.520189 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:30.520161    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:48:30.520665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:30.520637    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 00:48:31.520282 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:31.520250    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:48:31.520743 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:31.520487    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 00:48:32.519896 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:32.519860    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:48:32.520101 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:32.519933    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 00:48:33.519637 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:33.519598    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 31 00:48:33.520001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:33.519670    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 31 00:48:34.729750 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:34.729715    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:34.730613 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:34.730594    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:34.731496 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:34.731476    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:48:44.326364 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:44.326332    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"]
Oct 31 00:48:44.777371 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:44.777329    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:48:49.326795 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.326738    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk" podUID="8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" containerName="payload-pre-processing" containerID="cri-o://56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5" gracePeriod=25
Oct 31 00:48:49.616331 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.616307    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:48:49.679768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.679738    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume\") pod \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") "
Oct 31 00:48:49.679928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.679790    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv\" (UniqueName: \"kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv\") pod \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\" (UID: \"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb\") "
Oct 31 00:48:49.680183 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.680158    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume" pod "8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" (UID: "8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:48:49.682036 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.682014    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv" pod "8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" (UID: "8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb"). InnerVolumeSpecName "kube-api-access-f6kxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:48:49.800427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.783660    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:49.800427 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:49.783692    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6kxv\" (UniqueName: \"kubernetes.io/projected/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb-kube-api-access-f6kxv\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:48:50.351414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.351368    4478 generic.go:356] "Generic (PLEG): container finished" podID="8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" containerID="56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5" exitCode=0
Oct 31 00:48:50.351904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.351428    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"
Oct 31 00:48:50.351904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.351458    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk" event={"ID":"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb","Type":"ContainerDied","Data":"56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5"}
Oct 31 00:48:50.351904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.351527    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk" event={"ID":"8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb","Type":"ContainerDied","Data":"c8a3be257608ad9d698354d5d12f2ddfb7a28afdfb1d50c386272bc36bf02dad"}
Oct 31 00:48:50.351904 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.351557    4478 scope.go:122] "RemoveContainer" containerID="56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5"
Oct 31 00:48:50.361098 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.361083    4478 scope.go:122] "RemoveContainer" containerID="56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5"
Oct 31 00:48:50.361365 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:48:50.361345    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5\": container with ID starting with 56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5 not found: ID does not exist" containerID="56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5"
Oct 31 00:48:50.361464 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.361374    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5"} err="failed to get container status \"56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5\": rpc error: code = NotFound desc = could not find container \"56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5\": container with ID starting with 56ed448c22365d1ecf674c77fd0cb0cddc8cc33bb76f20b203640151e59120e5 not found: ID does not exist"
Oct 31 00:48:50.515345 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.515312    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"]
Oct 31 00:48:50.517415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:50.517365    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-31144e-75df7b549f-gzjqk"]
Oct 31 00:48:51.117751 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:48:51.117718    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" path="/var/lib/kubelet/pods/8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb/volumes"
Oct 31 00:49:05.103120 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:05.103083    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"]
Oct 31 00:49:10.104220 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.104179    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t" podUID="00f4823e-4db5-455a-90a6-2bc045f5257f" containerName="payload-processing" containerID="cri-o://3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77" gracePeriod=25
Oct 31 00:49:10.351022 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.350985    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:49:10.449677 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.449632    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r\" (UniqueName: \"kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r\") pod \"00f4823e-4db5-455a-90a6-2bc045f5257f\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") "
Oct 31 00:49:10.449855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.449734    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume\") pod \"00f4823e-4db5-455a-90a6-2bc045f5257f\" (UID: \"00f4823e-4db5-455a-90a6-2bc045f5257f\") "
Oct 31 00:49:10.450130 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.450107    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume" pod "00f4823e-4db5-455a-90a6-2bc045f5257f" (UID: "00f4823e-4db5-455a-90a6-2bc045f5257f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:10.452129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.452098    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r" pod "00f4823e-4db5-455a-90a6-2bc045f5257f" (UID: "00f4823e-4db5-455a-90a6-2bc045f5257f"). InnerVolumeSpecName "kube-api-access-fbt6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:10.494530 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.494496    4478 generic.go:356] "Generic (PLEG): container finished" podID="00f4823e-4db5-455a-90a6-2bc045f5257f" containerID="3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77" exitCode=0
Oct 31 00:49:10.494703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.494545    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"
Oct 31 00:49:10.494703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.494582    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t" event={"ID":"00f4823e-4db5-455a-90a6-2bc045f5257f","Type":"ContainerDied","Data":"3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77"}
Oct 31 00:49:10.494703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.494651    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t" event={"ID":"00f4823e-4db5-455a-90a6-2bc045f5257f","Type":"ContainerDied","Data":"118bc5f66e68e949841151896949926fbbad1bfc70187f78435be192cd998a9e"}
Oct 31 00:49:10.494703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.494681    4478 scope.go:122] "RemoveContainer" containerID="3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77"
Oct 31 00:49:10.508457 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.508052    4478 scope.go:122] "RemoveContainer" containerID="3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77"
Oct 31 00:49:10.508457 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:10.508431    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77\": container with ID starting with 3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77 not found: ID does not exist" containerID="3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77"
Oct 31 00:49:10.508611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.508463    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77"} err="failed to get container status \"3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77\": rpc error: code = NotFound desc = could not find container \"3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77\": container with ID starting with 3bb11ea9f6c1e4e3909ea0ec1f385699c10ba21aed282ef2546125e7bb05ec77 not found: ID does not exist"
Oct 31 00:49:10.551305 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.551276    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbt6r\" (UniqueName: \"kubernetes.io/projected/00f4823e-4db5-455a-90a6-2bc045f5257f-kube-api-access-fbt6r\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:10.551459 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.551309    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00f4823e-4db5-455a-90a6-2bc045f5257f-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:10.659268 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.659233    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"]
Oct 31 00:49:10.664158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:10.664134    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-bf38e4-78664cc9b5-2cx2t"]
Oct 31 00:49:11.117464 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:11.117432    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f4823e-4db5-455a-90a6-2bc045f5257f" path="/var/lib/kubelet/pods/00f4823e-4db5-455a-90a6-2bc045f5257f/volumes"
Oct 31 00:49:14.757029 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:14.756988    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:49:29.409089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:29.409053    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"]
Oct 31 00:49:29.824154 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:29.824117    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"]
Oct 31 00:49:29.824397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:29.824354    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" containerID="cri-o://6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452" gracePeriod=30
Oct 31 00:49:31.562740 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:31.562696    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"]
Oct 31 00:49:31.563246 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:31.562984    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" podUID="fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" containerName="maas-api" containerID="cri-o://e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206" gracePeriod=30
Oct 31 00:49:31.855477 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:31.855453    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:49:32.020302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.020262    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs\") pod \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") "
Oct 31 00:49:32.020508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.020334    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca\") pod \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") "
Oct 31 00:49:32.020508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.020458    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw\" (UniqueName: \"kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw\") pod \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") "
Oct 31 00:49:32.020508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.020498    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\") pod \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\" (UID: \"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0\") "
Oct 31 00:49:32.021858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.021829    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca" pod "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" (UID: "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:32.024144 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.024109    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs" pod "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" (UID: "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:32.045875 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.045827    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw" pod "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" (UID: "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0"). InnerVolumeSpecName "kube-api-access-2vhtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:32.047988 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.047797    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls" pod "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" (UID: "fa263a05-e0f4-47bc-a0e0-050dcaa06ff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:32.125088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.124217    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:32.125088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.124257    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:32.125088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.124274    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vhtw\" (UniqueName: \"kubernetes.io/projected/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-kube-api-access-2vhtw\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:32.125088 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.124289    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:32.615476 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.615438    4478 generic.go:356] "Generic (PLEG): container finished" podID="fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" containerID="e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206" exitCode=0
Oct 31 00:49:32.615837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.615498    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"
Oct 31 00:49:32.615837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.615518    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" event={"ID":"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0","Type":"ContainerDied","Data":"e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206"}
Oct 31 00:49:32.615837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.615590    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw" event={"ID":"fa263a05-e0f4-47bc-a0e0-050dcaa06ff0","Type":"ContainerDied","Data":"e9f69b7104f9e0806ef641d62994324671f789e9eab9948ad01682b0ff608cef"}
Oct 31 00:49:32.615837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.615614    4478 scope.go:122] "RemoveContainer" containerID="e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206"
Oct 31 00:49:32.625367 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.625352    4478 scope.go:122] "RemoveContainer" containerID="e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206"
Oct 31 00:49:32.625640 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:32.625623    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206\": container with ID starting with e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206 not found: ID does not exist" containerID="e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206"
Oct 31 00:49:32.625706 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.625647    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206"} err="failed to get container status \"e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206\": rpc error: code = NotFound desc = could not find container \"e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206\": container with ID starting with e0e09260e4c19e46cfcd47b69f021813d9e4fb0a94393f05696713a96af50206 not found: ID does not exist"
Oct 31 00:49:32.783012 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.782975    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"]
Oct 31 00:49:32.830790 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:32.830753    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-bf38e4-54bbff9f7d-vnfzw"]
Oct 31 00:49:33.118762 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:33.118729    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" path="/var/lib/kubelet/pods/fa263a05-e0f4-47bc-a0e0-050dcaa06ff0/volumes"
Oct 31 00:49:33.744060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:33.744026    4478 patch_prober.go:29] interesting pod/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:49:33.744472 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:33.744088    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:49:34.409823 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.409782    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9" podUID="bfa89316-903b-4485-bb66-6e882970bab7" containerName="payload-processing" containerID="cri-o://dca328233ec6a3299d8c8f2f4d42c10bbf3b9e7344baf81c5fbf3aae4f91c385" gracePeriod=25
Oct 31 00:49:34.630199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.630165    4478 generic.go:356] "Generic (PLEG): container finished" podID="bfa89316-903b-4485-bb66-6e882970bab7" containerID="dca328233ec6a3299d8c8f2f4d42c10bbf3b9e7344baf81c5fbf3aae4f91c385" exitCode=0
Oct 31 00:49:34.630406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.630278    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9" event={"ID":"bfa89316-903b-4485-bb66-6e882970bab7","Type":"ContainerDied","Data":"dca328233ec6a3299d8c8f2f4d42c10bbf3b9e7344baf81c5fbf3aae4f91c385"}
Oct 31 00:49:34.686601 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.686575    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:49:34.857479 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.857452    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume\") pod \"bfa89316-903b-4485-bb66-6e882970bab7\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") "
Oct 31 00:49:34.857849 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.857561    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh\" (UniqueName: \"kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh\") pod \"bfa89316-903b-4485-bb66-6e882970bab7\" (UID: \"bfa89316-903b-4485-bb66-6e882970bab7\") "
Oct 31 00:49:34.857938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.857912    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume" pod "bfa89316-903b-4485-bb66-6e882970bab7" (UID: "bfa89316-903b-4485-bb66-6e882970bab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:34.859848 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.859826    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh" pod "bfa89316-903b-4485-bb66-6e882970bab7" (UID: "bfa89316-903b-4485-bb66-6e882970bab7"). InnerVolumeSpecName "kube-api-access-h9xdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:34.959329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.959300    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9xdh\" (UniqueName: \"kubernetes.io/projected/bfa89316-903b-4485-bb66-6e882970bab7-kube-api-access-h9xdh\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:34.959473 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:34.959334    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfa89316-903b-4485-bb66-6e882970bab7-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.093871 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.093846    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:49:35.262789 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.262701    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.262955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.262814    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.262955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.262842    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.262955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.262874    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.262955 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.262916    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.263176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.262981    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.263176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263116    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.263176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263129    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:35.263176 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263161    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.263366 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263190    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket\") pod \"a695baf2-603d-4156-9404-3e92e926cf22\" (UID: \"a695baf2-603d-4156-9404-3e92e926cf22\") "
Oct 31 00:49:35.263443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263365    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:35.263443 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263421    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:35.263861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263636    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.263861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263658    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.263861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263672    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.263861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263828    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:35.264307 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.263950    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:35.266700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.265796    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:35.266700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.266053    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:35.267066 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.267038    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "kube-api-access-z7g77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:35.267162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.267091    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token" pod "a695baf2-603d-4156-9404-3e92e926cf22" (UID: "a695baf2-603d-4156-9404-3e92e926cf22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:35.364598 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.364564    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7g77\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-kube-api-access-z7g77\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.364598 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.364596    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a695baf2-603d-4156-9404-3e92e926cf22-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.364785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.364610    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a695baf2-603d-4156-9404-3e92e926cf22-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.364785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.364622    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a695baf2-603d-4156-9404-3e92e926cf22-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.364785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.364635    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.364785 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.364648    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a695baf2-603d-4156-9404-3e92e926cf22-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:35.638578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.638497    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9" event={"ID":"bfa89316-903b-4485-bb66-6e882970bab7","Type":"ContainerDied","Data":"4871b3b94ff2380c17a5f04543ca9ae7e7d138235fd5df5c4e8dd438ff40853b"}
Oct 31 00:49:35.638578 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.638558    4478 scope.go:122] "RemoveContainer" containerID="dca328233ec6a3299d8c8f2f4d42c10bbf3b9e7344baf81c5fbf3aae4f91c385"
Oct 31 00:49:35.638807 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.638501    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"
Oct 31 00:49:35.640760 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.640734    4478 generic.go:356] "Generic (PLEG): container finished" podID="a695baf2-603d-4156-9404-3e92e926cf22" containerID="6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452" exitCode=0
Oct 31 00:49:35.640905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.640778    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" event={"ID":"a695baf2-603d-4156-9404-3e92e926cf22","Type":"ContainerDied","Data":"6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452"}
Oct 31 00:49:35.640905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.640807    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"
Oct 31 00:49:35.640905 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.640822    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk" event={"ID":"a695baf2-603d-4156-9404-3e92e926cf22","Type":"ContainerDied","Data":"421ad87b3247bfc3b017f8d12ab47c2f055423d732e1b9c685c548b37985e9bd"}
Oct 31 00:49:35.647922 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.647902    4478 scope.go:122] "RemoveContainer" containerID="6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452"
Oct 31 00:49:35.657949 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.657928    4478 scope.go:122] "RemoveContainer" containerID="0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"
Oct 31 00:49:35.665855 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.665832    4478 scope.go:122] "RemoveContainer" containerID="6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452"
Oct 31 00:49:35.666124 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:35.666105    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452\": container with ID starting with 6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452 not found: ID does not exist" containerID="6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452"
Oct 31 00:49:35.666206 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.666132    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452"} err="failed to get container status \"6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452\": rpc error: code = NotFound desc = could not find container \"6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452\": container with ID starting with 6406449d63e79fa77319eeaf890ae36e325055ed0e68178bb99682c478573452 not found: ID does not exist"
Oct 31 00:49:35.666206 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.666147    4478 scope.go:122] "RemoveContainer" containerID="0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"
Oct 31 00:49:35.666469 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:35.666439    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519\": container with ID starting with 0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519 not found: ID does not exist" containerID="0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"
Oct 31 00:49:35.666568 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.666477    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519"} err="failed to get container status \"0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519\": rpc error: code = NotFound desc = could not find container \"0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519\": container with ID starting with 0178770d0aa9e7ceb528fcb91ad976686eb19bb7d8ca5a5da27f19fb65689519 not found: ID does not exist"
Oct 31 00:49:35.802645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.802608    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"]
Oct 31 00:49:35.805418 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.805371    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-31144e-openshift-default-5b4f97fc57-rv8kk"]
Oct 31 00:49:35.918608 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.918538    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"]
Oct 31 00:49:35.925276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:35.925253    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-868b06-c7c87d669-5htx9"]
Oct 31 00:49:37.118976 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:37.118948    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a695baf2-603d-4156-9404-3e92e926cf22" path="/var/lib/kubelet/pods/a695baf2-603d-4156-9404-3e92e926cf22/volumes"
Oct 31 00:49:37.119362 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:37.119295    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfa89316-903b-4485-bb66-6e882970bab7" path="/var/lib/kubelet/pods/bfa89316-903b-4485-bb66-6e882970bab7/volumes"
Oct 31 00:49:40.749108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:40.749071    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"]
Oct 31 00:49:40.749563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:40.749354    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy" containerID="cri-o://bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488" gracePeriod=30
Oct 31 00:49:41.781821 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.781783    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"]
Oct 31 00:49:41.782470 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782449    4478 cpu_manager.go:414] "containerMap: removing container" podUID="8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" containerName="payload-pre-processing"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782472    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" containerName="payload-pre-processing"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782486    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782494    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782514    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782521    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782531    4478 cpu_manager.go:414] "containerMap: removing container" podUID="00f4823e-4db5-455a-90a6-2bc045f5257f" containerName="payload-processing"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782538    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f4823e-4db5-455a-90a6-2bc045f5257f" containerName="payload-processing"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782555    4478 cpu_manager.go:414] "containerMap: removing container" podUID="bfa89316-903b-4485-bb66-6e882970bab7" containerName="payload-processing"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782561    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfa89316-903b-4485-bb66-6e882970bab7" containerName="payload-processing"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782581    4478 cpu_manager.go:414] "containerMap: removing container" podUID="fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" containerName="maas-api"
Oct 31 00:49:41.782590 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782588    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" containerName="maas-api"
Oct 31 00:49:41.783160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782693    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy"
Oct 31 00:49:41.783160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782704    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f4823e-4db5-455a-90a6-2bc045f5257f" containerName="payload-processing"
Oct 31 00:49:41.783160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782716    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfa89316-903b-4485-bb66-6e882970bab7" containerName="payload-processing"
Oct 31 00:49:41.783160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782728    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa263a05-e0f4-47bc-a0e0-050dcaa06ff0" containerName="maas-api"
Oct 31 00:49:41.783160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782742    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9dc7df-da5f-462f-99ab-2a13ed2a1cfb" containerName="payload-pre-processing"
Oct 31 00:49:41.783160 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.782750    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a695baf2-603d-4156-9404-3e92e926cf22" containerName="istio-proxy"
Oct 31 00:49:41.787711 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.787688    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.844861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.844830    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.844880    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.844923    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.844950    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845011 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.844978    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.845047    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845199 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.845173    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb7z5\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845265 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.845228    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.845313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.845263    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.846171 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.846148    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"]
Oct 31 00:49:41.946118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946084    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946125    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946153    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946274 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946263    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946282    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946326    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946353    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb7z5\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946409    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946452    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946818 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946791    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946813    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946861    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.946961 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.946900    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.947150 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.947128    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.948808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.948788    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:41.948901 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:41.948846    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:42.006688 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.006660    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:42.006881 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.006861    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb7z5\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5\") pod \"e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") " pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:42.101144 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.101056    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:42.537247 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.537210    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"]
Oct 31 00:49:42.575809 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:49:42.575783    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5480a159_4564_4f5a_9f63_23aa9c6f0708.slice/crio-0b85ab0db4c385d331c96cfe0253600dbfa6ada94d5ce6321e17cbc72d1690e4 WatchSource:0}: Error finding container 0b85ab0db4c385d331c96cfe0253600dbfa6ada94d5ce6321e17cbc72d1690e4: Status 404 returned error can't find the container with id 0b85ab0db4c385d331c96cfe0253600dbfa6ada94d5ce6321e17cbc72d1690e4
Oct 31 00:49:42.578064 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.578033    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:49:42.578153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.578099    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:49:42.578153 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.578125    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:49:42.683573 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.683535    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" event={"ID":"5480a159-4564-4f5a-9f63-23aa9c6f0708","Type":"ContainerStarted","Data":"f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1"}
Oct 31 00:49:42.683765 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:42.683630    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" event={"ID":"5480a159-4564-4f5a-9f63-23aa9c6f0708","Type":"ContainerStarted","Data":"0b85ab0db4c385d331c96cfe0253600dbfa6ada94d5ce6321e17cbc72d1690e4"}
Oct 31 00:49:43.101622 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:43.101589    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:43.103319 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:43.103293    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:43.103483 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:43.103344    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:44.101644 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:44.101610    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:44.102015 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:44.101661    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:44.781648 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:44.781610    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:49:45.105405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:45.103581    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:45.105405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:45.103641    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:46.054404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.044833    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:49:46.088569 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088533    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088729 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088643    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088729 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088719    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088755    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088868 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088810    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088898    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088945    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.088986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088973    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.089141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088993    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:46.089141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.088921    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:46.089141 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089028    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token\") pod \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\" (UID: \"3b4a155e-335a-4431-9d0c-1e57d02daf1a\") "
Oct 31 00:49:46.089299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089154    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:46.089299 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089222    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:46.089591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089539    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.089717 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089594    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.089717 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089605    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:46.089845 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089705    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.089845 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.089798    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.091224 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.091195    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:49:46.091332 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.091311    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "kube-api-access-gntb6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:46.091511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.091491    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:49:46.092163 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.092133    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token" pod "3b4a155e-335a-4431-9d0c-1e57d02daf1a" (UID: "3b4a155e-335a-4431-9d0c-1e57d02daf1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:46.101842 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.101822    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:46.101928 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.101865    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:46.191302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.191267    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.191302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.191293    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.191302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.191302    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.191302 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.191312    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b4a155e-335a-4431-9d0c-1e57d02daf1a-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.191859 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.191322    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gntb6\" (UniqueName: \"kubernetes.io/projected/3b4a155e-335a-4431-9d0c-1e57d02daf1a-kube-api-access-gntb6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:46.267368 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.267251    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podStartSLOduration=5.267235638 podStartE2EDuration="5.267235638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:42.804475214 +0000 UTC m=+2815.821139927" watchObservedRunningTime="2026-10-31 00:49:46.267235638 +0000 UTC m=+2819.283900351"
Oct 31 00:49:46.707490 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.707459    4478 generic.go:356] "Generic (PLEG): container finished" podID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerID="bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488" exitCode=0
Oct 31 00:49:46.707667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.707518    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"
Oct 31 00:49:46.707667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.707546    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" event={"ID":"3b4a155e-335a-4431-9d0c-1e57d02daf1a","Type":"ContainerDied","Data":"bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488"}
Oct 31 00:49:46.707667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.707610    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw" event={"ID":"3b4a155e-335a-4431-9d0c-1e57d02daf1a","Type":"ContainerDied","Data":"f4306cb2caf3a16c9f9fbec1a78d1ceb9562298acb928e9cbee5f6c2e7ef5569"}
Oct 31 00:49:46.707667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.707633    4478 scope.go:122] "RemoveContainer" containerID="bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488"
Oct 31 00:49:46.718936 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.718919    4478 scope.go:122] "RemoveContainer" containerID="2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"
Oct 31 00:49:46.728754 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.728738    4478 scope.go:122] "RemoveContainer" containerID="bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488"
Oct 31 00:49:46.729024 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:46.729006    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488\": container with ID starting with bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488 not found: ID does not exist" containerID="bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488"
Oct 31 00:49:46.729075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.729034    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488"} err="failed to get container status \"bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488\": rpc error: code = NotFound desc = could not find container \"bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488\": container with ID starting with bfee6a6eb7337255bd2d906573794f06f562c8a514359c4981abe72358ddb488 not found: ID does not exist"
Oct 31 00:49:46.729075 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.729052    4478 scope.go:122] "RemoveContainer" containerID="2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"
Oct 31 00:49:46.729297 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:46.729278    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366\": container with ID starting with 2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366 not found: ID does not exist" containerID="2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"
Oct 31 00:49:46.729344 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.729307    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366"} err="failed to get container status \"2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366\": rpc error: code = NotFound desc = could not find container \"2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366\": container with ID starting with 2a93c76a71b96be3643ff58e87673d79118c542f1dd992fcec5361235fa72366 not found: ID does not exist"
Oct 31 00:49:46.889884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.889848    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"]
Oct 31 00:49:46.897511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:46.897484    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-bf38e4-openshift-default-5d6c55845c-zg9gw"]
Oct 31 00:49:47.102292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:47.102207    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:47.102292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:47.102278    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:47.117667 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:47.117641    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" path="/var/lib/kubelet/pods/3b4a155e-335a-4431-9d0c-1e57d02daf1a/volumes"
Oct 31 00:49:48.084151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.081305    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"]
Oct 31 00:49:48.084151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.081722    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d" podUID="dd8de6ff-a085-482b-82d4-b95095092e43" containerName="maas-api" containerID="cri-o://f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b" gracePeriod=30
Oct 31 00:49:48.114687 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.114651    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:48.117001 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.114826    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:48.488225 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.488151    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:49:48.594172 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.594121    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\") pod \"dd8de6ff-a085-482b-82d4-b95095092e43\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") "
Oct 31 00:49:48.594361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.594199    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc\" (UniqueName: \"kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc\") pod \"dd8de6ff-a085-482b-82d4-b95095092e43\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") "
Oct 31 00:49:48.594361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.594261    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca\") pod \"dd8de6ff-a085-482b-82d4-b95095092e43\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") "
Oct 31 00:49:48.594361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.594290    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs\") pod \"dd8de6ff-a085-482b-82d4-b95095092e43\" (UID: \"dd8de6ff-a085-482b-82d4-b95095092e43\") "
Oct 31 00:49:48.595420 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.595342    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca" pod "dd8de6ff-a085-482b-82d4-b95095092e43" (UID: "dd8de6ff-a085-482b-82d4-b95095092e43"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:49:48.598460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.598420    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls" pod "dd8de6ff-a085-482b-82d4-b95095092e43" (UID: "dd8de6ff-a085-482b-82d4-b95095092e43"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:48.655407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.643514    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc" pod "dd8de6ff-a085-482b-82d4-b95095092e43" (UID: "dd8de6ff-a085-482b-82d4-b95095092e43"). InnerVolumeSpecName "kube-api-access-tjsjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:49:48.655407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.643897    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs" pod "dd8de6ff-a085-482b-82d4-b95095092e43" (UID: "dd8de6ff-a085-482b-82d4-b95095092e43"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 00:49:48.697405 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.697347    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:48.697587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.697410    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjsjc\" (UniqueName: \"kubernetes.io/projected/dd8de6ff-a085-482b-82d4-b95095092e43-kube-api-access-tjsjc\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:48.697587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.697427    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dd8de6ff-a085-482b-82d4-b95095092e43-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:48.697587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.697441    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd8de6ff-a085-482b-82d4-b95095092e43-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:49:48.735014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.734203    4478 generic.go:356] "Generic (PLEG): container finished" podID="dd8de6ff-a085-482b-82d4-b95095092e43" containerID="f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b" exitCode=0
Oct 31 00:49:48.735014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.734334    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d" event={"ID":"dd8de6ff-a085-482b-82d4-b95095092e43","Type":"ContainerDied","Data":"f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b"}
Oct 31 00:49:48.735014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.734405    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d" event={"ID":"dd8de6ff-a085-482b-82d4-b95095092e43","Type":"ContainerDied","Data":"5f0cd6806b4ca6f17b8cd0820e02c77067d06ad3cd78e0bd11eab9d757c17f88"}
Oct 31 00:49:48.735014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.734442    4478 scope.go:122] "RemoveContainer" containerID="f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b"
Oct 31 00:49:48.735014 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.735002    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"
Oct 31 00:49:48.764982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.763630    4478 scope.go:122] "RemoveContainer" containerID="f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b"
Oct 31 00:49:48.764982 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:49:48.763967    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b\": container with ID starting with f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b not found: ID does not exist" containerID="f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b"
Oct 31 00:49:48.764982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.764004    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b"} err="failed to get container status \"f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b\": rpc error: code = NotFound desc = could not find container \"f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b\": container with ID starting with f22683ebf650f8fcdf15b12b7234fdaa00fcba662e8e89517eccf3331a8f460b not found: ID does not exist"
Oct 31 00:49:48.916645 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.916600    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"]
Oct 31 00:49:48.920832 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:48.920799    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-868b06-945964778-x958d"]
Oct 31 00:49:49.106947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:49.104337    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 31 00:49:49.106947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:49.104478    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 31 00:49:49.122755 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:49.122716    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd8de6ff-a085-482b-82d4-b95095092e43" path="/var/lib/kubelet/pods/dd8de6ff-a085-482b-82d4-b95095092e43/volumes"
Oct 31 00:49:50.105460 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:50.105435    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:50.105775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:50.105748    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:50.106433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:50.106412    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 00:49:53.041148 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.041082    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"]
Oct 31 00:49:53.042463 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.042435    4478 cpu_manager.go:414] "containerMap: removing container" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy"
Oct 31 00:49:53.042628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.042615    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy"
Oct 31 00:49:53.042722 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.042713    4478 cpu_manager.go:414] "containerMap: removing container" podUID="dd8de6ff-a085-482b-82d4-b95095092e43" containerName="maas-api"
Oct 31 00:49:53.042808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.042787    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd8de6ff-a085-482b-82d4-b95095092e43" containerName="maas-api"
Oct 31 00:49:53.043038 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.043020    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy"
Oct 31 00:49:53.043124 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.043047    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd8de6ff-a085-482b-82d4-b95095092e43" containerName="maas-api"
Oct 31 00:49:53.048781 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.048720    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.057415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.052431    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj7x2\" (UniqueName: \"kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2\") pod \"payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.057415 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.052484    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.102576 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.102546    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"]
Oct 31 00:49:53.149951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.149916    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-cfc055-dockercfg-wcx4n\""
Oct 31 00:49:53.149951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.149923    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-cfc055\""
Oct 31 00:49:53.153551 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.153528    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj7x2\" (UniqueName: \"kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2\") pod \"payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.153639 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.153567    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.154263 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.154245    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.214549 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.214518    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj7x2\" (UniqueName: \"kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2\") pod \"payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.383238 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.383144    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:53.769374 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.769343    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"]
Oct 31 00:49:53.773612 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:49:53.773578    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0bec69_d5ef_496c_90d0_831f8a37651d.slice/crio-23ce0877f4b4b947a3a53acf9d626a1a13f6d7774b960cbe1812a111c8fdd67d WatchSource:0}: Error finding container 23ce0877f4b4b947a3a53acf9d626a1a13f6d7774b960cbe1812a111c8fdd67d: Status 404 returned error can't find the container with id 23ce0877f4b4b947a3a53acf9d626a1a13f6d7774b960cbe1812a111c8fdd67d
Oct 31 00:49:53.791902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:53.791878    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" event={"ID":"da0bec69-d5ef-496c-90d0-831f8a37651d","Type":"ContainerStarted","Data":"23ce0877f4b4b947a3a53acf9d626a1a13f6d7774b960cbe1812a111c8fdd67d"}
Oct 31 00:49:54.799049 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:54.799010    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" event={"ID":"da0bec69-d5ef-496c-90d0-831f8a37651d","Type":"ContainerStarted","Data":"0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc"}
Oct 31 00:49:54.799450 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:54.799080    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:49:54.915995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:54.915940    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" podStartSLOduration=2.9159271540000002 podStartE2EDuration="2.915927154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:54.914517089 +0000 UTC m=+2827.931181844" watchObservedRunningTime="2026-10-31 00:49:54.915927154 +0000 UTC m=+2827.932591865"
Oct 31 00:49:58.043069 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.043027    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"]
Oct 31 00:49:58.043597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.043573    4478 cpu_manager.go:414] "containerMap: removing container" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy"
Oct 31 00:49:58.043597 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.043589    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy"
Oct 31 00:49:58.043717 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.043662    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4a155e-335a-4431-9d0c-1e57d02daf1a" containerName="istio-proxy"
Oct 31 00:49:58.046943 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.046921    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.100149 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.100096    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-f13553\""
Oct 31 00:49:58.100292 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.100259    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-f13553-dockercfg-2vtct\""
Oct 31 00:49:58.104804 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.104777    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"]
Oct 31 00:49:58.125447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.113409    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.125447 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.113466    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf6l8\" (UniqueName: \"kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8\") pod \"payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.194045 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.194013    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"]
Oct 31 00:49:58.199414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.199392    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.215034 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.215005    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.215175 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.215062    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf6l8\" (UniqueName: \"kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8\") pod \"payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.216499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.216478    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.255520 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.255492    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"]
Oct 31 00:49:58.276408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.276349    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf6l8\" (UniqueName: \"kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8\") pod \"payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.315926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.315861    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npz9v\" (UniqueName: \"kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v\") pod \"payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.315926 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.315908    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.363947 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.363907    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:49:58.419207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.417598    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npz9v\" (UniqueName: \"kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v\") pod \"payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.419207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.417801    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.419207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.418935    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.480323 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.480295    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npz9v\" (UniqueName: \"kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v\") pod \"payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.512920 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.512885    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:58.742549 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.742523    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"]
Oct 31 00:49:58.745212 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:49:58.745181    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda237c6d0_a8f8_4af0_84b6_82610978e14a.slice/crio-93a8e3f187f1be9ff476320970f6647ebeeefd382ebc7b2c558bfd2ddca5180e WatchSource:0}: Error finding container 93a8e3f187f1be9ff476320970f6647ebeeefd382ebc7b2c558bfd2ddca5180e: Status 404 returned error can't find the container with id 93a8e3f187f1be9ff476320970f6647ebeeefd382ebc7b2c558bfd2ddca5180e
Oct 31 00:49:58.823168 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.823139    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" event={"ID":"a237c6d0-a8f8-4af0-84b6-82610978e14a","Type":"ContainerStarted","Data":"93a8e3f187f1be9ff476320970f6647ebeeefd382ebc7b2c558bfd2ddca5180e"}
Oct 31 00:49:58.888836 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:58.888811    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"]
Oct 31 00:49:58.891574 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:49:58.891540    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c211c5_56f7_411b_8d89_f293640b8bd5.slice/crio-0f37fde1cb5dc7a6ef6da261fff7ee243880123a32e1e90f38e884b2a5005cf0 WatchSource:0}: Error finding container 0f37fde1cb5dc7a6ef6da261fff7ee243880123a32e1e90f38e884b2a5005cf0: Status 404 returned error can't find the container with id 0f37fde1cb5dc7a6ef6da261fff7ee243880123a32e1e90f38e884b2a5005cf0
Oct 31 00:49:59.835407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:59.835246    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" event={"ID":"c1c211c5-56f7-411b-8d89-f293640b8bd5","Type":"ContainerStarted","Data":"56833486f8904a204135ee30077c1a3026918b8eebc1f9cac753ebc3d21a3f16"}
Oct 31 00:49:59.835407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:59.835309    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" event={"ID":"c1c211c5-56f7-411b-8d89-f293640b8bd5","Type":"ContainerStarted","Data":"0f37fde1cb5dc7a6ef6da261fff7ee243880123a32e1e90f38e884b2a5005cf0"}
Oct 31 00:49:59.835407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:59.835360    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:49:59.842434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:59.837819    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" event={"ID":"a237c6d0-a8f8-4af0-84b6-82610978e14a","Type":"ContainerStarted","Data":"237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b"}
Oct 31 00:49:59.842434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:49:59.838313    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:50:00.084681 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:00.082649    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" podStartSLOduration=2.082627159 podStartE2EDuration="2.082627159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:59.960838512 +0000 UTC m=+2832.977503225" watchObservedRunningTime="2026-10-31 00:50:00.082627159 +0000 UTC m=+2833.099291871"
Oct 31 00:50:05.807833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:05.807805    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 00:50:05.930924 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:05.930874    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" podStartSLOduration=8.93085935 podStartE2EDuration="8.93085935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:00.089211018 +0000 UTC m=+2833.105875746" watchObservedRunningTime="2026-10-31 00:50:05.93085935 +0000 UTC m=+2838.947524092"
Oct 31 00:50:09.975684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:09.975648    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"]
Oct 31 00:50:09.979967 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:09.979944    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.024862 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.024832    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.025007 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.024986    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjflt\" (UniqueName: \"kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt\") pod \"payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.034060 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.033968    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-16b984\""
Oct 31 00:50:10.085414 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.083756    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-16b984-dockercfg-pf7nl\""
Oct 31 00:50:10.089410 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.089173    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"]
Oct 31 00:50:10.127604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.126060    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.127604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.126194    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjflt\" (UniqueName: \"kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt\") pod \"payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.128455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.128423    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.136467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.136443    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"]
Oct 31 00:50:10.140397 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.140353    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.187257 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.187229    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjflt\" (UniqueName: \"kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt\") pod \"payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.198634 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.198609    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"]
Oct 31 00:50:10.227580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.227515    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7cnh\" (UniqueName: \"kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh\") pod \"payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.227580 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.227561    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.294542 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.294505    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:10.329164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.329134    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7cnh\" (UniqueName: \"kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh\") pod \"payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.329319 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.329186    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.329872 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.329853    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.391276 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.391240    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7cnh\" (UniqueName: \"kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh\") pod \"payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.453437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.453366    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.677296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.677268    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"]
Oct 31 00:50:10.679361 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:50:10.679328    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod884be0a1_f3b8_4450_95a0_23de11cf4033.slice/crio-875da3b99280b1a339eb26e371eecda8bbf947cc3f330d3b4a4d1b754413bb90 WatchSource:0}: Error finding container 875da3b99280b1a339eb26e371eecda8bbf947cc3f330d3b4a4d1b754413bb90: Status 404 returned error can't find the container with id 875da3b99280b1a339eb26e371eecda8bbf947cc3f330d3b4a4d1b754413bb90
Oct 31 00:50:10.834431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.834403    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"]
Oct 31 00:50:10.836764 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:50:10.836738    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205c9ed4_35da_4f91_b0dc_738abddef884.slice/crio-8a7ce71a11db931e2fe81710e4727f9648eb6f4110fac6f47ff0a39c3ff5a08a WatchSource:0}: Error finding container 8a7ce71a11db931e2fe81710e4727f9648eb6f4110fac6f47ff0a39c3ff5a08a: Status 404 returned error can't find the container with id 8a7ce71a11db931e2fe81710e4727f9648eb6f4110fac6f47ff0a39c3ff5a08a
Oct 31 00:50:10.908887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.908855    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 00:50:10.962102 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.962068    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" event={"ID":"205c9ed4-35da-4f91-b0dc-738abddef884","Type":"ContainerStarted","Data":"ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892"}
Oct 31 00:50:10.962277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.962137    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:10.962277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.962156    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" event={"ID":"205c9ed4-35da-4f91-b0dc-738abddef884","Type":"ContainerStarted","Data":"8a7ce71a11db931e2fe81710e4727f9648eb6f4110fac6f47ff0a39c3ff5a08a"}
Oct 31 00:50:10.963703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.963681    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" event={"ID":"884be0a1-f3b8-4450-95a0-23de11cf4033","Type":"ContainerStarted","Data":"d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82"}
Oct 31 00:50:10.963829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.963720    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" event={"ID":"884be0a1-f3b8-4450-95a0-23de11cf4033","Type":"ContainerStarted","Data":"875da3b99280b1a339eb26e371eecda8bbf947cc3f330d3b4a4d1b754413bb90"}
Oct 31 00:50:10.963829 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:10.963742    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:11.141588 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:11.141493    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" podStartSLOduration=1.141481521 podStartE2EDuration="1.141481521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:11.139072149 +0000 UTC m=+2844.155736876" watchObservedRunningTime="2026-10-31 00:50:11.141481521 +0000 UTC m=+2844.158146232"
Oct 31 00:50:11.270614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:11.270542    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" podStartSLOduration=2.270517861 podStartE2EDuration="2.270517861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:11.261644563 +0000 UTC m=+2844.278309346" watchObservedRunningTime="2026-10-31 00:50:11.270517861 +0000 UTC m=+2844.287182584"
Oct 31 00:50:11.915081 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:11.915053    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 00:50:14.760583 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:50:14.760544    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:50:21.972277 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:21.972244    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 00:50:21.972814 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:21.972316    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 00:50:25.214555 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.214511    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"]
Oct 31 00:50:25.221425 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.219752    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.283334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.283303    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"]
Oct 31 00:50:25.327494 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.327312    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-f13553-serving-cert\""
Oct 31 00:50:25.328313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.328289    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.328478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.328423    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.328478 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.328456    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.328614 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.328485    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmql\" (UniqueName: \"kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.429887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.429858    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.429887 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.429890    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.430132 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.429914    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmql\" (UniqueName: \"kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.430132 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.429995    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.430132 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:50:25.430012    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-serving-cert: secret "maas-api-e2e-shared-a-w3-f13553-serving-cert" not found
Oct 31 00:50:25.430132 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:50:25.430097    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls podName:dc5105fc-4fac-432d-a0b7-c85a6b17e532 nodeName:}" failed. No retries permitted until 2026-10-31 00:50:25.930076649 +0000 UTC m=+2858.946741356 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls") pod "maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" (UID: "dc5105fc-4fac-432d-a0b7-c85a6b17e532") : secret "maas-api-e2e-shared-a-w3-f13553-serving-cert" not found
Oct 31 00:50:25.430700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.430681    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.432464 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.432445    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.492409 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.492293    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmql\" (UniqueName: \"kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.935980 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.935950    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:25.938455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:25.938430    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:26.139392 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:26.139361    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:26.517164 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:26.517124    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"]
Oct 31 00:50:26.519415 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:50:26.519375    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5105fc_4fac_432d_a0b7_c85a6b17e532.slice/crio-ec1c5a6dc9540feb0396ceae59d32dde6d685d68c8a795d08e88fe807409bad5 WatchSource:0}: Error finding container ec1c5a6dc9540feb0396ceae59d32dde6d685d68c8a795d08e88fe807409bad5: Status 404 returned error can't find the container with id ec1c5a6dc9540feb0396ceae59d32dde6d685d68c8a795d08e88fe807409bad5
Oct 31 00:50:26.520870 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:26.520852    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 00:50:27.051764 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:27.051722    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" event={"ID":"dc5105fc-4fac-432d-a0b7-c85a6b17e532","Type":"ContainerStarted","Data":"ec1c5a6dc9540feb0396ceae59d32dde6d685d68c8a795d08e88fe807409bad5"}
Oct 31 00:50:28.059465 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:28.059419    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" event={"ID":"dc5105fc-4fac-432d-a0b7-c85a6b17e532","Type":"ContainerStarted","Data":"3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff"}
Oct 31 00:50:28.059882 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:28.059667    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:28.177982 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:28.177925    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" podStartSLOduration=2.342892676 podStartE2EDuration="3.177909429s" totalImagesPullingTime="835.016753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:28.176160428 +0000 UTC m=+2861.192825159" watchObservedRunningTime="2026-10-31 00:50:28.177909429 +0000 UTC m=+2861.194574140"
Oct 31 00:50:34.071495 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:34.071462    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 00:50:35.955892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:35.955856    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"]
Oct 31 00:50:35.960703 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:35.960685    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.014351 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.014319    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-4743f8\""
Oct 31 00:50:36.068434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.063055    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4b8f\" (UniqueName: \"kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f\") pod \"payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.068434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.063202    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.068434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.063736    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-4743f8-dockercfg-87t8l\""
Oct 31 00:50:36.068434 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.068311    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"]
Oct 31 00:50:36.116714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.116679    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"]
Oct 31 00:50:36.121304 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.121281    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.164508 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.164474    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh7n6\" (UniqueName: \"kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6\") pod \"payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.164700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.164660    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.164783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.164716    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4b8f\" (UniqueName: \"kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f\") pod \"payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.164783 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.164764    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.165600 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.165572    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.166086 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.166059    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"]
Oct 31 00:50:36.226204 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.226137    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4b8f\" (UniqueName: \"kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f\") pod \"payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.266275 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.266234    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh7n6\" (UniqueName: \"kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6\") pod \"payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.266492 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.266341    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.267256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.267228    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.273610 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.273590    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:36.328524 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.328490    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh7n6\" (UniqueName: \"kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6\") pod \"payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.435243 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.435206    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:36.650963 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.650905    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"]
Oct 31 00:50:36.653486 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:50:36.653455    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1689d832_f216_4ebc_aa4e_650155a07944.slice/crio-1cc6f15f1da12c32accad54e694f5f7a35e111d47e67648c30bebe4355bd6e4b WatchSource:0}: Error finding container 1cc6f15f1da12c32accad54e694f5f7a35e111d47e67648c30bebe4355bd6e4b: Status 404 returned error can't find the container with id 1cc6f15f1da12c32accad54e694f5f7a35e111d47e67648c30bebe4355bd6e4b
Oct 31 00:50:36.813647 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:36.813623    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"]
Oct 31 00:50:36.815557 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:50:36.815529    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ef74ae_1614_4a6c_aa76_a9761c27d98c.slice/crio-8304b236c469d7a8c914f04915822d535ce63931ba99ff9f1c82ce53b89517ff WatchSource:0}: Error finding container 8304b236c469d7a8c914f04915822d535ce63931ba99ff9f1c82ce53b89517ff: Status 404 returned error can't find the container with id 8304b236c469d7a8c914f04915822d535ce63931ba99ff9f1c82ce53b89517ff
Oct 31 00:50:37.118129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.118041    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" event={"ID":"1689d832-f216-4ebc-aa4e-650155a07944","Type":"ContainerStarted","Data":"52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a"}
Oct 31 00:50:37.118129 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.118098    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" event={"ID":"1689d832-f216-4ebc-aa4e-650155a07944","Type":"ContainerStarted","Data":"1cc6f15f1da12c32accad54e694f5f7a35e111d47e67648c30bebe4355bd6e4b"}
Oct 31 00:50:37.118650 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.118197    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:37.119632 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.119607    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" event={"ID":"07ef74ae-1614-4a6c-aa76-a9761c27d98c","Type":"ContainerStarted","Data":"7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea"}
Oct 31 00:50:37.119756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.119642    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" event={"ID":"07ef74ae-1614-4a6c-aa76-a9761c27d98c","Type":"ContainerStarted","Data":"8304b236c469d7a8c914f04915822d535ce63931ba99ff9f1c82ce53b89517ff"}
Oct 31 00:50:37.119756 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.119662    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:37.234826 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.234779    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" podStartSLOduration=2.234766029 podStartE2EDuration="2.234766029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:37.233261225 +0000 UTC m=+2870.249925965" watchObservedRunningTime="2026-10-31 00:50:37.234766029 +0000 UTC m=+2870.251430782"
Oct 31 00:50:37.363006 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:37.362943    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" podStartSLOduration=1.362922485 podStartE2EDuration="1.362922485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:37.362772995 +0000 UTC m=+2870.379437709" watchObservedRunningTime="2026-10-31 00:50:37.362922485 +0000 UTC m=+2870.379587198"
Oct 31 00:50:44.759129 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:50:44.759090    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:50:45.825313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.825284    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"]
Oct 31 00:50:45.830295 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.830279    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:45.884407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.883978    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-16b984-serving-cert\""
Oct 31 00:50:45.902499 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.902469    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf6dc\" (UniqueName: \"kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:45.902656 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.902516    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:45.902730 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.902676    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:45.902792 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.902730    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:45.971404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:45.970550    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"]
Oct 31 00:50:46.003938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.003890    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.004134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.003988    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf6dc\" (UniqueName: \"kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.004134 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.004024    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.004261 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.004176    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.004354 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:50:46.004334    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-serving-cert: secret "maas-api-e2e-shared-a-w2-16b984-serving-cert" not found
Oct 31 00:50:46.004482 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:50:46.004460    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls podName:057c7160-c16b-4af3-bc50-c6719be2af5f nodeName:}" failed. No retries permitted until 2026-10-31 00:50:46.504437247 +0000 UTC m=+2879.521101978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" (UID: "057c7160-c16b-4af3-bc50-c6719be2af5f") : secret "maas-api-e2e-shared-a-w2-16b984-serving-cert" not found
Oct 31 00:50:46.005639 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.005614    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.008435 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.007804    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.065131 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.065102    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf6dc\" (UniqueName: \"kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.363516 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.363480    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"]
Oct 31 00:50:46.367814 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.367791    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.421361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.421338    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-1a1181\""
Oct 31 00:50:46.421742 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.421720    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-1a1181-dockercfg-6fww2\""
Oct 31 00:50:46.424669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.424646    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"]
Oct 31 00:50:46.518151 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.518110    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.518320 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.518193    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.518320 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.518274    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z\") pod \"payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.520700 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.520676    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:46.619808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.619730    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.619953 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.619824    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z\") pod \"payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.620489 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.620467    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.680830 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.680797    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z\") pod \"payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.681567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.681544    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:46.751139 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:46.750676    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:47.063921 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.063895    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"]
Oct 31 00:50:47.064324 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:50:47.063917    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1ad11bf_61cf_4b3e_9db7_f4130428de57.slice/crio-6d43f7434b422befc028a577f7879628d97026f224796bb5ca15222966ed92ec WatchSource:0}: Error finding container 6d43f7434b422befc028a577f7879628d97026f224796bb5ca15222966ed92ec: Status 404 returned error can't find the container with id 6d43f7434b422befc028a577f7879628d97026f224796bb5ca15222966ed92ec
Oct 31 00:50:47.175797 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.175771    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"]
Oct 31 00:50:47.218241 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.218210    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt" event={"ID":"b1ad11bf-61cf-4b3e-9db7-f4130428de57","Type":"ContainerStarted","Data":"b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f"}
Oct 31 00:50:47.218375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.218270    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt" event={"ID":"b1ad11bf-61cf-4b3e-9db7-f4130428de57","Type":"ContainerStarted","Data":"6d43f7434b422befc028a577f7879628d97026f224796bb5ca15222966ed92ec"}
Oct 31 00:50:47.218375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.218301    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:50:47.219468 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.219446    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" event={"ID":"057c7160-c16b-4af3-bc50-c6719be2af5f","Type":"ContainerStarted","Data":"a772e6ed16fc5f8a85f6d3f3532332dfacc6321561945bb1380208716f35afbd"}
Oct 31 00:50:47.334752 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:47.334653    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt" podStartSLOduration=1.33463566 podStartE2EDuration="1.33463566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:47.333195047 +0000 UTC m=+2880.349859782" watchObservedRunningTime="2026-10-31 00:50:47.33463566 +0000 UTC m=+2880.351300374"
Oct 31 00:50:48.132202 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:48.132165    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 00:50:48.132775 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:48.132753    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 00:50:48.227149 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:48.227115    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" event={"ID":"057c7160-c16b-4af3-bc50-c6719be2af5f","Type":"ContainerStarted","Data":"ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9"}
Oct 31 00:50:48.227488 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:48.227456    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:48.536913 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:48.536832    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" podStartSLOduration=2.70766783 podStartE2EDuration="3.536812815s" totalImagesPullingTime="829.144985ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:48.534678342 +0000 UTC m=+2881.551343064" watchObservedRunningTime="2026-10-31 00:50:48.536812815 +0000 UTC m=+2881.553477528"
Oct 31 00:50:54.238906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:54.238873    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 00:50:58.230016 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:50:58.229989    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:51:14.758462 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:51:14.758427    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:51:19.681244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.681210    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"]
Oct 31 00:51:19.685593 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.685573    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.815664 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.815609    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"]
Oct 31 00:51:19.827628 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.827596    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-4e1a6c-openshift-default-dockercfg-xcnzj\""
Oct 31 00:51:19.908137 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908099    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908341 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908155    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908341 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908224    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908341 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908259    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908342    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908372    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908465    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf97\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908567 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908488    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:19.908790 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:19.908574    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009259    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf97\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009308    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009413 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009374    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009525    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009584    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009612    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009643    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009714 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009697    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009725    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009888    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.009995 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.009965    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.010185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.010050    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.010185 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.010164    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.010626 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.010603    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.012027 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.012004    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.012689 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.012663    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.070253 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.070227    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.070376 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.070354    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf97\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97\") pod \"e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") " pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.298565 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.298466    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:20.783902 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.783877    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"]
Oct 31 00:51:20.786197 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:51:20.786171    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dd3e4c3_4a5d_412e_aa59_320547bc5c6a.slice/crio-ac886f7e148860f939a68e9f11e9d521afd166f303c802426f163a7b6a7de33f WatchSource:0}: Error finding container ac886f7e148860f939a68e9f11e9d521afd166f303c802426f163a7b6a7de33f: Status 404 returned error can't find the container with id ac886f7e148860f939a68e9f11e9d521afd166f303c802426f163a7b6a7de33f
Oct 31 00:51:20.788642 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.788609    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:51:20.788748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.788688    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:51:20.788748 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:20.788735    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:51:21.426363 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:21.426317    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" event={"ID":"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a","Type":"ContainerStarted","Data":"5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"}
Oct 31 00:51:21.426556 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:21.426402    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" event={"ID":"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a","Type":"ContainerStarted","Data":"ac886f7e148860f939a68e9f11e9d521afd166f303c802426f163a7b6a7de33f"}
Oct 31 00:51:21.552568 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:21.552502    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podStartSLOduration=2.552484632 podStartE2EDuration="2.552484632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:21.550873231 +0000 UTC m=+2914.567537955" watchObservedRunningTime="2026-10-31 00:51:21.552484632 +0000 UTC m=+2914.569149411"
Oct 31 00:51:22.298952 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:22.298921    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:22.306222 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:22.306194    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:22.431831 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:22.431795    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:22.432769 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:22.432750    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:51:26.461951 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.461911    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"]
Oct 31 00:51:26.466524 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.466500    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.495041 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.495017    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.495158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.495050    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn\") pod \"payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.532665 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.532640    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"]
Oct 31 00:51:26.606456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.584693    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c6b3b9\""
Oct 31 00:51:26.606456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.584875    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c6b3b9-dockercfg-fcrjn\""
Oct 31 00:51:26.606456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.597072    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.606456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.597112    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn\") pod \"payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.606456 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.598359    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.690211 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.690178    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn\") pod \"payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:26.783222 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:26.783138    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:27.162661 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:27.162635    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"]
Oct 31 00:51:27.164919 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:51:27.164891    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod990aab48_fa5c_48d2_be02_6c48dd0431a6.slice/crio-ae190ab53991514d143a798593686fec02b24217770c5f9b09d157f0b5944689 WatchSource:0}: Error finding container ae190ab53991514d143a798593686fec02b24217770c5f9b09d157f0b5944689: Status 404 returned error can't find the container with id ae190ab53991514d143a798593686fec02b24217770c5f9b09d157f0b5944689
Oct 31 00:51:27.464693 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:27.464654    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" event={"ID":"990aab48-fa5c-48d2-be02-6c48dd0431a6","Type":"ContainerStarted","Data":"86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7"}
Oct 31 00:51:27.465111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:27.464721    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" event={"ID":"990aab48-fa5c-48d2-be02-6c48dd0431a6","Type":"ContainerStarted","Data":"ae190ab53991514d143a798593686fec02b24217770c5f9b09d157f0b5944689"}
Oct 31 00:51:27.465111 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:27.464780    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:27.582063 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:27.582012    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" podStartSLOduration=1.581998222 podStartE2EDuration="1.581998222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:27.579173125 +0000 UTC m=+2920.595837854" watchObservedRunningTime="2026-10-31 00:51:27.581998222 +0000 UTC m=+2920.598662934"
Oct 31 00:51:31.346808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.346723    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"]
Oct 31 00:51:31.350912 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.350894    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.490406 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.453799    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4e1a6c\""
Oct 31 00:51:31.491353 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.458297    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4e1a6c-dockercfg-hvfhj\""
Oct 31 00:51:31.491563 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.463349    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"]
Oct 31 00:51:31.491719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.491283    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np2vb\" (UniqueName: \"kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb\") pod \"payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.491719 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.491712    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.532417 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.494189    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"]
Oct 31 00:51:31.556808 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.556781    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.561618 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.561570    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"]
Oct 31 00:51:31.602118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.594114    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np2vb\" (UniqueName: \"kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb\") pod \"payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.602118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.594221    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.602118 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.595302    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.667493 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.667465    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np2vb\" (UniqueName: \"kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb\") pod \"payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:31.695475 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.695442    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblbj\" (UniqueName: \"kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj\") pod \"payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.695620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.695506    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.801105 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.799246    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblbj\" (UniqueName: \"kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj\") pod \"payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.805861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.799373    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.805861 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.804908    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.864711 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.864621    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblbj\" (UniqueName: \"kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj\") pod \"payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.915751 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.915716    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:31.965062 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:31.965030    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:32.303823 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.303727    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"]
Oct 31 00:51:32.306486 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:51:32.306461    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a6da29_6c28_4e85_ac93_2ee7f518516e.slice/crio-7b3b24d7acc9a0e5793e522d28efb88a2f986d56f2d9d29b5b19a1686a8f439c WatchSource:0}: Error finding container 7b3b24d7acc9a0e5793e522d28efb88a2f986d56f2d9d29b5b19a1686a8f439c: Status 404 returned error can't find the container with id 7b3b24d7acc9a0e5793e522d28efb88a2f986d56f2d9d29b5b19a1686a8f439c
Oct 31 00:51:32.368611 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.368584    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"]
Oct 31 00:51:32.369918 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:51:32.369889    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8aacb8_6c17_4ba6_a440_cff64a9f85bc.slice/crio-56d997924855e81d007c932b80a99aad1ca229f1196ad0071f8bbe63db32bb56 WatchSource:0}: Error finding container 56d997924855e81d007c932b80a99aad1ca229f1196ad0071f8bbe63db32bb56: Status 404 returned error can't find the container with id 56d997924855e81d007c932b80a99aad1ca229f1196ad0071f8bbe63db32bb56
Oct 31 00:51:32.610837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.610740    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" event={"ID":"65a6da29-6c28-4e85-ac93-2ee7f518516e","Type":"ContainerStarted","Data":"79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159"}
Oct 31 00:51:32.610837 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.610805    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" event={"ID":"65a6da29-6c28-4e85-ac93-2ee7f518516e","Type":"ContainerStarted","Data":"7b3b24d7acc9a0e5793e522d28efb88a2f986d56f2d9d29b5b19a1686a8f439c"}
Oct 31 00:51:32.611074 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.610851    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:32.612433 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.612409    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" event={"ID":"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc","Type":"ContainerStarted","Data":"bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848"}
Oct 31 00:51:32.612574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.612450    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" event={"ID":"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc","Type":"ContainerStarted","Data":"56d997924855e81d007c932b80a99aad1ca229f1196ad0071f8bbe63db32bb56"}
Oct 31 00:51:32.612574 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.612542    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:32.727455 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.727403    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" podStartSLOduration=1.727373322 podStartE2EDuration="1.727373322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:32.725722929 +0000 UTC m=+2925.742387653" watchObservedRunningTime="2026-10-31 00:51:32.727373322 +0000 UTC m=+2925.744038036"
Oct 31 00:51:32.854404 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:32.846821    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" podStartSLOduration=1.846798784 podStartE2EDuration="1.846798784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:32.843856779 +0000 UTC m=+2925.860521509" watchObservedRunningTime="2026-10-31 00:51:32.846798784 +0000 UTC m=+2925.863463497"
Oct 31 00:51:38.491676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:38.491646    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 00:51:43.621108 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:43.621068    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 00:51:43.621511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:43.621164    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 00:51:44.756200 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:51:44.756166    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:51:48.481178 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.481139    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"]
Oct 31 00:51:48.494421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.486336    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.550408 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.541733    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-bf325c-openshift-default-dockercfg-ms7qj\""
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611472    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611514    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611582    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcvf5\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611605    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611623    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611657    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611681    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611723    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.613824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.611746    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.618073 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.617804    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"]
Oct 31 00:51:48.712933 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.712895    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.712958    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713034    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713066    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713214 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713194    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713242    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713305    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcvf5\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713336    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.713587 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.713367    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.714084 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.714047    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.714084 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.714058    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.714224 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.714204    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.714362 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.714334    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.714718 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.714572    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.716421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.716273    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.716421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.716330    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.774089 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.774012    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcvf5\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.774226 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.774164    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token\") pod \"e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") " pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:48.861407 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:48.860211    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:49.308594 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.308563    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"]
Oct 31 00:51:49.310510 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:51:49.310478    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94c221e4_e004_428e_bc7c_de47bf336acc.slice/crio-c2cc0109eab8b2f9a6413f937c19f45f3f11bd4293fe1e8b72e58cd80e6ea03e WatchSource:0}: Error finding container c2cc0109eab8b2f9a6413f937c19f45f3f11bd4293fe1e8b72e58cd80e6ea03e: Status 404 returned error can't find the container with id c2cc0109eab8b2f9a6413f937c19f45f3f11bd4293fe1e8b72e58cd80e6ea03e
Oct 31 00:51:49.313162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.313131    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:51:49.313244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.313195    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:51:49.313244 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.313222    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:51:49.722446 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.718514    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" event={"ID":"94c221e4-e004-428e-bc7c-de47bf336acc","Type":"ContainerStarted","Data":"67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"}
Oct 31 00:51:49.722446 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.718592    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" event={"ID":"94c221e4-e004-428e-bc7c-de47bf336acc","Type":"ContainerStarted","Data":"c2cc0109eab8b2f9a6413f937c19f45f3f11bd4293fe1e8b72e58cd80e6ea03e"}
Oct 31 00:51:49.838906 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.838847    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podStartSLOduration=1.838829751 podStartE2EDuration="1.838829751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:51:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:51:49.837681868 +0000 UTC m=+2942.854346572" watchObservedRunningTime="2026-10-31 00:51:49.838829751 +0000 UTC m=+2942.855494467"
Oct 31 00:51:49.860674 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:49.860642    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:50.866449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:50.866420    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:51.729933 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:51.729899    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:51:51.730938 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:51:51.730915    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:52:14.759455 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:52:14.759402    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:52:31.683607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.683567    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"]
Oct 31 00:52:31.687547 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.687531    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:31.741669 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.741638    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:31.741833 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.741715    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shm6r\" (UniqueName: \"kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r\") pod \"payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:31.742473 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.742446    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-093220\""
Oct 31 00:52:31.825237 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.823307    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"]
Oct 31 00:52:31.825712 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.825689    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-093220-dockercfg-h8jk6\""
Oct 31 00:52:31.842762 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.842727    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shm6r\" (UniqueName: \"kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r\") pod \"payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:31.843207 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.843184    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:31.844364 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.844338    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:31.909087 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:31.909052    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shm6r\" (UniqueName: \"kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r\") pod \"payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:32.001570 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:32.001492    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:32.419596 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:32.419572    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"]
Oct 31 00:52:32.421261 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:52:32.421231    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdb50a63_262a_4955_9b9b_3a1ea2ad2cc9.slice/crio-61ed8ef2d20782a9416edb2b2ea89ea128577da15247539f84c9dba858baff74 WatchSource:0}: Error finding container 61ed8ef2d20782a9416edb2b2ea89ea128577da15247539f84c9dba858baff74: Status 404 returned error can't find the container with id 61ed8ef2d20782a9416edb2b2ea89ea128577da15247539f84c9dba858baff74
Oct 31 00:52:33.080200 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:33.080162    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" event={"ID":"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9","Type":"ContainerStarted","Data":"e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f"}
Oct 31 00:52:33.080631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:33.080222    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" event={"ID":"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9","Type":"ContainerStarted","Data":"61ed8ef2d20782a9416edb2b2ea89ea128577da15247539f84c9dba858baff74"}
Oct 31 00:52:33.080631 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:33.080329    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:33.234421 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:33.218335    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" podStartSLOduration=2.218312755 podStartE2EDuration="2.218312755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:52:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:52:33.214008167 +0000 UTC m=+2986.230672903" watchObservedRunningTime="2026-10-31 00:52:33.218312755 +0000 UTC m=+2986.234977469"
Oct 31 00:52:44.089077 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:44.089040    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 00:52:44.789261 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:52:44.789223    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:52:52.215908 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.215873    4478 generic.go:356] "Generic (PLEG): container finished" podID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerID="5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75" exitCode=0
Oct 31 00:52:52.216330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.215945    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" event={"ID":"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a","Type":"ContainerDied","Data":"5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"}
Oct 31 00:52:52.216330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.216320    4478 scope.go:122] "RemoveContainer" containerID="5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"
Oct 31 00:52:52.217930 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.217906    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:52:52.218013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.217988    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:52:52.218051 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.218020    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:52:52.432076 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:52.432044    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:52:53.224338 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:53.224306    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" event={"ID":"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a","Type":"ContainerStarted","Data":"c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032"}
Oct 31 00:52:53.299361 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:53.299324    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:52:53.300607 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:53.300575    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 00:52:53.300745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:53.300633    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 00:52:54.299162 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:54.299128    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 00:52:54.299604 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:54.299186    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 00:52:55.299449 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:55.299413    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 00:52:55.299860 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:55.299462    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 00:52:56.299585 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:56.299544    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 00:52:56.300000 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:56.299616    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 00:52:57.299375 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:57.299336    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 00:52:57.299570 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:57.299405    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 00:52:58.298934 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:58.298901    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused" start-of-body=
Oct 31 00:52:58.299329 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:58.298958    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.140:15021/healthz/ready\": dial tcp 10.132.0.140:15021: connect: connection refused"
Oct 31 00:52:59.303453 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:59.303421    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:52:59.303932 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:59.303692    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:52:59.304315 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:52:59.304297    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 00:53:01.227984 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.227950    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"]
Oct 31 00:53:01.230826 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.230811    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.245110 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.245085    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dz6s5\" (UniqueName: \"kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.245221 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.245125    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.245296 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.245259    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.245334 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.245308    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.282854 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.282828    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-bf325c-serving-cert\""
Oct 31 00:53:01.336314 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.336288    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"]
Oct 31 00:53:01.346032 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.346007    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.346311 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:53:01.346287    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-serving-cert: secret "maas-api-e2e-shared-b-w2-bf325c-serving-cert" not found
Oct 31 00:53:01.346467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.346314    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dz6s5\" (UniqueName: \"kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.346467 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:53:01.346353    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls podName:7a7cc49d-8d85-4098-82aa-c1ec6c2cd029 nodeName:}" failed. No retries permitted until 2026-10-31 00:53:01.846332346 +0000 UTC m=+3014.862997059 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls") pod "maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" (UID: "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029") : secret "maas-api-e2e-shared-b-w2-bf325c-serving-cert" not found
Oct 31 00:53:01.346467 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.346432    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.346656 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.346604    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.347208 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.347183    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.349170 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.349149    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.407670 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.407641    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dz6s5\" (UniqueName: \"kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.855959 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.855924    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:01.858676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:01.858640    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:02.143313 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:02.143233    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:02.546620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:02.546590    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"]
Oct 31 00:53:02.547395 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:53:02.547349    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7cc49d_8d85_4098_82aa_c1ec6c2cd029.slice/crio-ebf99eb44895f95bedb1c7df53d1d9ed5c137ee7199f56dec8502abe8b380729 WatchSource:0}: Error finding container ebf99eb44895f95bedb1c7df53d1d9ed5c137ee7199f56dec8502abe8b380729: Status 404 returned error can't find the container with id ebf99eb44895f95bedb1c7df53d1d9ed5c137ee7199f56dec8502abe8b380729
Oct 31 00:53:03.296745 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:03.296708    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" event={"ID":"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029","Type":"ContainerStarted","Data":"ebf99eb44895f95bedb1c7df53d1d9ed5c137ee7199f56dec8502abe8b380729"}
Oct 31 00:53:04.310466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:04.308988    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" event={"ID":"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029","Type":"ContainerStarted","Data":"17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481"}
Oct 31 00:53:04.310466 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:04.309260    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:04.428810 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:04.428756    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" podStartSLOduration=2.560419739 podStartE2EDuration="3.428742601s" totalImagesPullingTime="868.322862ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:04.425990742 +0000 UTC m=+3017.442655455" watchObservedRunningTime="2026-10-31 00:53:04.428742601 +0000 UTC m=+3017.445407312"
Oct 31 00:53:10.321256 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:10.321220    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 00:53:14.760445 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:53:14.760404    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:53:35.019948 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.019908    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"]
Oct 31 00:53:35.023078 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.023058    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.075016 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.074991    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-093220-serving-cert\""
Oct 31 00:53:35.088728 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.088697    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z66gt\" (UniqueName: \"kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.088885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.088749    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.088885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.088796    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.088885 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.088827    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.129158 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.129047    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"]
Oct 31 00:53:35.190056 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.190017    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z66gt\" (UniqueName: \"kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.190228 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.190095    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.190228 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.190154    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.190228 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.190206    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.190455 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:53:35.190338    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-serving-cert: secret "maas-api-e2e-shared-b-w4-093220-serving-cert" not found
Oct 31 00:53:35.190455 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:53:35.190437    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls podName:695c4a13-794d-42da-a8ed-a88e0a19610e nodeName:}" failed. No retries permitted until 2026-10-31 00:53:35.690411182 +0000 UTC m=+3048.707075890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls") pod "maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" (UID: "695c4a13-794d-42da-a8ed-a88e0a19610e") : secret "maas-api-e2e-shared-b-w4-093220-serving-cert" not found
Oct 31 00:53:35.191068 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.191039    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.193243 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.193222    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.250765 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.250735    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z66gt\" (UniqueName: \"kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.700231 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.700192    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.702702 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.702671    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:35.938504 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:35.938474    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:36.333694 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:36.333661    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"]
Oct 31 00:53:36.337496 ip-10-0-137-148 kubenswrapper[4478]: W1031 00:53:36.337456    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod695c4a13_794d_42da_a8ed_a88e0a19610e.slice/crio-6fb6c85f4451a89c66ad567ff3af610c56815daf16a7f95a1785c1307b69b741 WatchSource:0}: Error finding container 6fb6c85f4451a89c66ad567ff3af610c56815daf16a7f95a1785c1307b69b741: Status 404 returned error can't find the container with id 6fb6c85f4451a89c66ad567ff3af610c56815daf16a7f95a1785c1307b69b741
Oct 31 00:53:36.505662 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:36.505628    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" event={"ID":"695c4a13-794d-42da-a8ed-a88e0a19610e","Type":"ContainerStarted","Data":"6fb6c85f4451a89c66ad567ff3af610c56815daf16a7f95a1785c1307b69b741"}
Oct 31 00:53:37.515771 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:37.515675    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" event={"ID":"695c4a13-794d-42da-a8ed-a88e0a19610e","Type":"ContainerStarted","Data":"42da25b5dce2b9a4be24a3cf47d3815f5113980153cce0c4a90dfd796dbc01a9"}
Oct 31 00:53:37.516154 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:37.515771    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:37.653117 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:37.653047    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" podStartSLOduration=2.7628793849999997 podStartE2EDuration="3.653027614s" totalImagesPullingTime="890.148229ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:53:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:53:37.649767723 +0000 UTC m=+3050.666432438" watchObservedRunningTime="2026-10-31 00:53:37.653027614 +0000 UTC m=+3050.669692331"
Oct 31 00:53:37.776591 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:37.776503    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 31 00:53:37.776844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:37.776814    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 31 00:53:38.523713 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:38.523652    4478 generic.go:356] "Generic (PLEG): container finished" podID="94c221e4-e004-428e-bc7c-de47bf336acc" containerID="67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4" exitCode=0
Oct 31 00:53:38.524114 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:38.523720    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" event={"ID":"94c221e4-e004-428e-bc7c-de47bf336acc","Type":"ContainerDied","Data":"67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"}
Oct 31 00:53:38.524182 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:38.524167    4478 scope.go:122] "RemoveContainer" containerID="67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"
Oct 31 00:53:38.525684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:38.525654    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:53:38.525768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:38.525750    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:53:38.525818 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:38.525802    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 00:53:39.539892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:39.539857    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" event={"ID":"94c221e4-e004-428e-bc7c-de47bf336acc","Type":"ContainerStarted","Data":"ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac"}
Oct 31 00:53:39.861127 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:39.861024    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:53:39.861350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:39.861283    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:53:39.861350 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:39.861328    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 00:53:40.861043 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:40.861006    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:53:40.861566 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:40.861068    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 00:53:41.861355 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:41.861320    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:53:41.861874 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:41.861410    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 00:53:42.861469 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:42.861437    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:53:42.861892 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:42.861490    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 00:53:43.529595 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:43.529565    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 00:53:43.862749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:43.862663    4478 patch_prober.go:29] interesting pod/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 31 00:53:43.862749 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:43.862720    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 31 00:53:44.762055 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:53:44.762016    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:53:44.866824 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:44.866735    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:53:44.867290 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:44.867143    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:53:44.867994 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:44.867972    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 00:53:59.960023 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:53:59.959123    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"]
Oct 31 00:54:04.960620 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:04.960579    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt" podUID="b1ad11bf-61cf-4b3e-9db7-f4130428de57" containerName="payload-pre-processing" containerID="cri-o://b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f" gracePeriod=25
Oct 31 00:54:05.229844 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.229821    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:54:05.306862 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.306833    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z\") pod \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") "
Oct 31 00:54:05.307027 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.306953    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume\") pod \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\" (UID: \"b1ad11bf-61cf-4b3e-9db7-f4130428de57\") "
Oct 31 00:54:05.307338 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.307309    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume" pod "b1ad11bf-61cf-4b3e-9db7-f4130428de57" (UID: "b1ad11bf-61cf-4b3e-9db7-f4130428de57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:54:05.309126 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.309109    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z" pod "b1ad11bf-61cf-4b3e-9db7-f4130428de57" (UID: "b1ad11bf-61cf-4b3e-9db7-f4130428de57"). InnerVolumeSpecName "kube-api-access-g5q8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:54:05.408820 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.408783    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b1ad11bf-61cf-4b3e-9db7-f4130428de57-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:54:05.408820 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.408813    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5q8z\" (UniqueName: \"kubernetes.io/projected/b1ad11bf-61cf-4b3e-9db7-f4130428de57-kube-api-access-g5q8z\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:54:05.725210 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.725171    4478 generic.go:356] "Generic (PLEG): container finished" podID="b1ad11bf-61cf-4b3e-9db7-f4130428de57" containerID="b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f" exitCode=0
Oct 31 00:54:05.725210 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.725213    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"
Oct 31 00:54:05.725858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.725258    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt" event={"ID":"b1ad11bf-61cf-4b3e-9db7-f4130428de57","Type":"ContainerDied","Data":"b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f"}
Oct 31 00:54:05.725858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.725315    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt" event={"ID":"b1ad11bf-61cf-4b3e-9db7-f4130428de57","Type":"ContainerDied","Data":"6d43f7434b422befc028a577f7879628d97026f224796bb5ca15222966ed92ec"}
Oct 31 00:54:05.725858 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.725346    4478 scope.go:122] "RemoveContainer" containerID="b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f"
Oct 31 00:54:05.735289 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.735272    4478 scope.go:122] "RemoveContainer" containerID="b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f"
Oct 31 00:54:05.735572 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:54:05.735547    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f\": container with ID starting with b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f not found: ID does not exist" containerID="b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f"
Oct 31 00:54:05.735676 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.735579    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f"} err="failed to get container status \"b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f\": rpc error: code = NotFound desc = could not find container \"b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f\": container with ID starting with b40c12fcbbd0b53e10137d54e9e79e9a18bc1261d2a1af8fe2f30eec0480bf7f not found: ID does not exist"
Oct 31 00:54:05.888684 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.888651    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"]
Oct 31 00:54:05.938250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:05.938215    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-1a1181-84cc549df9-ql2tt"]
Oct 31 00:54:07.119586 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:54:07.119553    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1ad11bf-61cf-4b3e-9db7-f4130428de57" path="/var/lib/kubelet/pods/b1ad11bf-61cf-4b3e-9db7-f4130428de57/volumes"
Oct 31 00:54:14.763275 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:54:14.763226    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:54:44.758162 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:54:44.758128    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:55:14.762617 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:55:14.762572    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:55:44.757540 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:55:44.757500    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:55:59.741389 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:55:59.741298    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"]
Oct 31 00:55:59.741898 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:55:59.741529    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" containerID="cri-o://62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921" gracePeriod=30
Oct 31 00:56:04.733425 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:04.733356    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 00:56:04.733816 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:04.733481    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 00:56:05.023666 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.023643    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:56:05.119264 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119234    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119294    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119322    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119343    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119371    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119437 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119432    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119493    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:05.119658 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119574    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119682    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:05.119768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119700    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:05.119768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119708    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119768 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119754    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy\") pod \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\" (UID: \"0bfa2710-f806-49cc-99dc-ae521f1c60ca\") "
Oct 31 00:56:05.119975 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119778    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 00:56:05.119975 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.119896    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:05.120511 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.120487    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.120690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.120664    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.120690 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.120692    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.120884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.120710    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.120884 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.120725    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.121888 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.121865    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "kube-api-access-6x9t5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:05.121986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.121898    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 00:56:05.121986 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.121913    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 00:56:05.122103 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.122023    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy" pod "0bfa2710-f806-49cc-99dc-ae521f1c60ca" (UID: "0bfa2710-f806-49cc-99dc-ae521f1c60ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 00:56:05.222013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.221974    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.222013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.222008    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x9t5\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-kube-api-access-6x9t5\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.222013 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.222021    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.222250 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.222032    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bfa2710-f806-49cc-99dc-ae521f1c60ca-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 00:56:05.481123 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.481088    4478 generic.go:356] "Generic (PLEG): container finished" podID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerID="62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921" exitCode=0
Oct 31 00:56:05.481330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.481148    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" event={"ID":"0bfa2710-f806-49cc-99dc-ae521f1c60ca","Type":"ContainerDied","Data":"62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921"}
Oct 31 00:56:05.481330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.481144    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"
Oct 31 00:56:05.481330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.481202    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf" event={"ID":"0bfa2710-f806-49cc-99dc-ae521f1c60ca","Type":"ContainerDied","Data":"0e40910e8d3c6ec162bd928b210c1972eae263efaf36dc9a80550ebfc8582067"}
Oct 31 00:56:05.481330 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.481230    4478 scope.go:122] "RemoveContainer" containerID="62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921"
Oct 31 00:56:05.494939 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.494909    4478 scope.go:122] "RemoveContainer" containerID="62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921"
Oct 31 00:56:05.495309 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:56:05.495280    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921\": container with ID starting with 62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921 not found: ID does not exist" containerID="62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921"
Oct 31 00:56:05.495431 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.495320    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921"} err="failed to get container status \"62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921\": rpc error: code = NotFound desc = could not find container \"62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921\": container with ID starting with 62a13f99c03aa97396c9dbf3f5fac5f2a5e7edfc4d1859535b119a36f5a7b921 not found: ID does not exist"
Oct 31 00:56:05.649279 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.649246    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"]
Oct 31 00:56:05.656545 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:05.656515    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-lk5xf"]
Oct 31 00:56:07.120390 ip-10-0-137-148 kubenswrapper[4478]: I1031 00:56:07.120356    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" path="/var/lib/kubelet/pods/0bfa2710-f806-49cc-99dc-ae521f1c60ca/volumes"
Oct 31 00:56:14.768288 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:56:14.768248    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:56:44.762723 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:56:44.762685    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:57:14.760274 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:57:14.760234    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:57:44.758870 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:57:44.758829    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:58:14.756825 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:58:14.756790    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:58:44.758472 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:58:44.758433    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:59:14.761244 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:59:14.761155    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 00:59:44.760036 ip-10-0-137-148 kubenswrapper[4478]: E1031 00:59:44.757839    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:00:11.931372 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.931338    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-2ac3b3"]
Oct 31 01:00:11.931944 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.931928    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b1ad11bf-61cf-4b3e-9db7-f4130428de57" containerName="payload-pre-processing"
Oct 31 01:00:11.931990 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.931945    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1ad11bf-61cf-4b3e-9db7-f4130428de57" containerName="payload-pre-processing"
Oct 31 01:00:11.931990 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.931967    4478 cpu_manager.go:414] "containerMap: removing container" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy"
Oct 31 01:00:11.931990 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.931972    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy"
Oct 31 01:00:11.932090 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.932051    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bfa2710-f806-49cc-99dc-ae521f1c60ca" containerName="istio-proxy"
Oct 31 01:00:11.932090 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.932065    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1ad11bf-61cf-4b3e-9db7-f4130428de57" containerName="payload-pre-processing"
Oct 31 01:00:11.935464 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.935449    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:11.975891 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.975860    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swsv4\" (UniqueName: \"kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4\") pod \"test-curl-7498-2ac3b3\" (UID: \"1dbd5510-99aa-416f-a3a0-201aa2a528e4\") " pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:11.991520 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:11.991487    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-2ac3b3"]
Oct 31 01:00:12.037207 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:12.037176    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\""
Oct 31 01:00:12.077451 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:12.077419    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swsv4\" (UniqueName: \"kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4\") pod \"test-curl-7498-2ac3b3\" (UID: \"1dbd5510-99aa-416f-a3a0-201aa2a528e4\") " pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:12.138140 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:12.138114    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swsv4\" (UniqueName: \"kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4\") pod \"test-curl-7498-2ac3b3\" (UID: \"1dbd5510-99aa-416f-a3a0-201aa2a528e4\") " pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:12.247493 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:12.247420    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:12.624714 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:12.624679    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-2ac3b3"]
Oct 31 01:00:12.629431 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:12.629400    4478 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 31 01:00:13.005646 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:13.005596    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-2ac3b3" event={"ID":"1dbd5510-99aa-416f-a3a0-201aa2a528e4","Type":"ContainerStarted","Data":"13f48c4de1ba3fd9c2488b18fa8d0dd17a8a15e11adb3db4116e4fe3ff57f1d1"}
Oct 31 01:00:14.760375 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:14.760335    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:00:20.050909 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:20.050866    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-2ac3b3" event={"ID":"1dbd5510-99aa-416f-a3a0-201aa2a528e4","Type":"ContainerStarted","Data":"fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb"}
Oct 31 01:00:20.166431 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:20.166353    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7498-2ac3b3" podStartSLOduration=2.401404738 podStartE2EDuration="9.16633605s" totalImagesPullingTime="6.764931312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:20.16618557 +0000 UTC m=+3453.182850283" watchObservedRunningTime="2026-10-31 01:00:20.16633605 +0000 UTC m=+3453.183000763"
Oct 31 01:00:21.643320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:21.643287    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-2ac3b3"]
Oct 31 01:00:21.649473 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:21.649444    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-2ac3b3"]
Oct 31 01:00:22.063319 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:22.063271    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7498-2ac3b3" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" containerName="test-curl-7498-2ac3b3" containerID="cri-o://fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb" gracePeriod=2
Oct 31 01:00:22.114425 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:22.114366    4478 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-2ac3b3\" is forbidden: User \"system:node:ip-10-0-137-148.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.us-west-1.compute.internal' and this object" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:24.307163 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:24.307135    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:24.422066 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:24.422000    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4\" (UniqueName: \"kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4\") pod \"1dbd5510-99aa-416f-a3a0-201aa2a528e4\" (UID: \"1dbd5510-99aa-416f-a3a0-201aa2a528e4\") "
Oct 31 01:00:24.424279 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:24.424249    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4" pod "1dbd5510-99aa-416f-a3a0-201aa2a528e4" (UID: "1dbd5510-99aa-416f-a3a0-201aa2a528e4"). InnerVolumeSpecName "kube-api-access-swsv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:24.522897 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:24.522867    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swsv4\" (UniqueName: \"kubernetes.io/projected/1dbd5510-99aa-416f-a3a0-201aa2a528e4-kube-api-access-swsv4\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:25.091863 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:25.091831    4478 generic.go:356] "Generic (PLEG): container finished" podID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" containerID="fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb" exitCode=137
Oct 31 01:00:25.092043 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:25.091873    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:25.092043 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:25.091925    4478 scope.go:122] "RemoveContainer" containerID="fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb"
Oct 31 01:00:25.101563 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:25.101548    4478 scope.go:122] "RemoveContainer" containerID="fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb"
Oct 31 01:00:25.101831 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:25.101802    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb\": container with ID starting with fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb not found: ID does not exist" containerID="fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb"
Oct 31 01:00:25.101883 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:25.101841    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb"} err="failed to get container status \"fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb\": rpc error: code = NotFound desc = could not find container \"fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb\": container with ID starting with fff43168a3154a5b73cd4a76797b57ee6a93c239fd3b468bbfda5c21cc282bdb not found: ID does not exist"
Oct 31 01:00:25.120443 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:25.120417    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" path="/var/lib/kubelet/pods/1dbd5510-99aa-416f-a3a0-201aa2a528e4/volumes"
Oct 31 01:00:25.151591 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:25.151560    4478 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-2ac3b3\" is forbidden: User \"system:node:ip-10-0-137-148.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.us-west-1.compute.internal' and this object" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" pod="opendatahub/test-curl-7498-2ac3b3"
Oct 31 01:00:26.308034 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.307998    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-655f87"]
Oct 31 01:00:26.309501 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.309473    4478 cpu_manager.go:414] "containerMap: removing container" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" containerName="test-curl-7498-2ac3b3"
Oct 31 01:00:26.309680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.309664    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" containerName="test-curl-7498-2ac3b3"
Oct 31 01:00:26.309994 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.309972    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dbd5510-99aa-416f-a3a0-201aa2a528e4" containerName="test-curl-7498-2ac3b3"
Oct 31 01:00:26.316193 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.316171    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:26.392921 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.392883    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-655f87"]
Oct 31 01:00:26.423310 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.423275    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\""
Oct 31 01:00:26.475857 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.475816    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmk5c\" (UniqueName: \"kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c\") pod \"test-curl-7498-655f87\" (UID: \"33a2905c-d4bf-4b41-90ae-31657a779726\") " pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:26.577570 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.577481    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmk5c\" (UniqueName: \"kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c\") pod \"test-curl-7498-655f87\" (UID: \"33a2905c-d4bf-4b41-90ae-31657a779726\") " pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:26.645307 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.645276    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmk5c\" (UniqueName: \"kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c\") pod \"test-curl-7498-655f87\" (UID: \"33a2905c-d4bf-4b41-90ae-31657a779726\") " pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:26.932837 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:26.932807    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:27.311929 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:27.311845    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-655f87"]
Oct 31 01:00:27.314996 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:00:27.314970    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33a2905c_d4bf_4b41_90ae_31657a779726.slice/crio-c910ec25d5fd138327cf393c7a67af978345d2af2944135089a382e63bb99c73 WatchSource:0}: Error finding container c910ec25d5fd138327cf393c7a67af978345d2af2944135089a382e63bb99c73: Status 404 returned error can't find the container with id c910ec25d5fd138327cf393c7a67af978345d2af2944135089a382e63bb99c73
Oct 31 01:00:28.115884 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:28.115840    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-655f87" event={"ID":"33a2905c-d4bf-4b41-90ae-31657a779726","Type":"ContainerStarted","Data":"f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00"}
Oct 31 01:00:28.115884 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:28.115887    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-655f87" event={"ID":"33a2905c-d4bf-4b41-90ae-31657a779726","Type":"ContainerStarted","Data":"c910ec25d5fd138327cf393c7a67af978345d2af2944135089a382e63bb99c73"}
Oct 31 01:00:28.230611 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:28.230559    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7498-655f87" podStartSLOduration=2.230545492 podStartE2EDuration="2.230545492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:28.228547255 +0000 UTC m=+3461.245211990" watchObservedRunningTime="2026-10-31 01:00:28.230545492 +0000 UTC m=+3461.247210203"
Oct 31 01:00:29.670967 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.670933    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-655f87"]
Oct 31 01:00:29.675977 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.675950    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-655f87"]
Oct 31 01:00:29.836558 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.836520    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7498-13087b"]
Oct 31 01:00:29.837473 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.837452    4478 cpu_manager.go:414] "containerMap: removing container" podUID="33a2905c-d4bf-4b41-90ae-31657a779726" containerName="test-curl-7498-655f87"
Oct 31 01:00:29.837522 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.837478    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33a2905c-d4bf-4b41-90ae-31657a779726" containerName="test-curl-7498-655f87"
Oct 31 01:00:29.837647 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.837633    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="33a2905c-d4bf-4b41-90ae-31657a779726" containerName="test-curl-7498-655f87"
Oct 31 01:00:29.841595 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.841577    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:29.894668 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.894638    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-13087b"]
Oct 31 01:00:29.921223 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:29.921141    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vms7g\" (UniqueName: \"kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g\") pod \"test-curl-7498-13087b\" (UID: \"da736741-54b9-404b-906f-7f779cd3a9bd\") " pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:30.022584 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:30.022554    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vms7g\" (UniqueName: \"kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g\") pod \"test-curl-7498-13087b\" (UID: \"da736741-54b9-404b-906f-7f779cd3a9bd\") " pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:30.083465 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:30.083439    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vms7g\" (UniqueName: \"kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g\") pod \"test-curl-7498-13087b\" (UID: \"da736741-54b9-404b-906f-7f779cd3a9bd\") " pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:30.127010 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:30.126971    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7498-655f87" podUID="33a2905c-d4bf-4b41-90ae-31657a779726" containerName="test-curl-7498-655f87" containerID="cri-o://f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00" gracePeriod=2
Oct 31 01:00:30.153532 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:30.153507    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:30.178143 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:30.178111    4478 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-655f87\" is forbidden: User \"system:node:ip-10-0-137-148.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.us-west-1.compute.internal' and this object" podUID="33a2905c-d4bf-4b41-90ae-31657a779726" pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:30.542577 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:30.542547    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7498-13087b"]
Oct 31 01:00:30.545609 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:00:30.545568    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda736741_54b9_404b_906f_7f779cd3a9bd.slice/crio-ef00505972c768d38f743dc3390389b3da2e115c019f4abdab8e2a43e5d8ec86 WatchSource:0}: Error finding container ef00505972c768d38f743dc3390389b3da2e115c019f4abdab8e2a43e5d8ec86: Status 404 returned error can't find the container with id ef00505972c768d38f743dc3390389b3da2e115c019f4abdab8e2a43e5d8ec86
Oct 31 01:00:31.133703 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.133664    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-13087b" event={"ID":"da736741-54b9-404b-906f-7f779cd3a9bd","Type":"ContainerStarted","Data":"24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7"}
Oct 31 01:00:31.133703 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.133700    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7498-13087b" event={"ID":"da736741-54b9-404b-906f-7f779cd3a9bd","Type":"ContainerStarted","Data":"ef00505972c768d38f743dc3390389b3da2e115c019f4abdab8e2a43e5d8ec86"}
Oct 31 01:00:31.248817 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.248769    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7498-13087b" podStartSLOduration=2.248755635 podStartE2EDuration="2.248755635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:31.246656316 +0000 UTC m=+3464.263321028" watchObservedRunningTime="2026-10-31 01:00:31.248755635 +0000 UTC m=+3464.265420349"
Oct 31 01:00:31.907872 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.907829    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"]
Oct 31 01:00:31.908111 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.908082    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" containerID="cri-o://5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055" gracePeriod=30
Oct 31 01:00:31.957993 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.957958    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"]
Oct 31 01:00:31.958234 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:31.958207    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" containerID="cri-o://f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1" gracePeriod=30
Oct 31 01:00:32.350543 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:32.350512    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:32.454393 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:32.454301    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c\" (UniqueName: \"kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c\") pod \"33a2905c-d4bf-4b41-90ae-31657a779726\" (UID: \"33a2905c-d4bf-4b41-90ae-31657a779726\") "
Oct 31 01:00:32.456474 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:32.456359    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c" pod "33a2905c-d4bf-4b41-90ae-31657a779726" (UID: "33a2905c-d4bf-4b41-90ae-31657a779726"). InnerVolumeSpecName "kube-api-access-rmk5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:32.555323 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:32.555292    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmk5c\" (UniqueName: \"kubernetes.io/projected/33a2905c-d4bf-4b41-90ae-31657a779726-kube-api-access-rmk5c\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:32.788918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:32.788828    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7498-13087b"]
Oct 31 01:00:32.793028 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:32.793001    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7498-13087b"]
Oct 31 01:00:33.118671 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.118597    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33a2905c-d4bf-4b41-90ae-31657a779726" path="/var/lib/kubelet/pods/33a2905c-d4bf-4b41-90ae-31657a779726/volumes"
Oct 31 01:00:33.166354 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.166317    4478 generic.go:356] "Generic (PLEG): container finished" podID="33a2905c-d4bf-4b41-90ae-31657a779726" containerID="f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00" exitCode=137
Oct 31 01:00:33.166556 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.166362    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-655f87"
Oct 31 01:00:33.166556 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.166411    4478 scope.go:122] "RemoveContainer" containerID="f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00"
Oct 31 01:00:33.166691 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.166576    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7498-13087b" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" containerName="test-curl-7498-13087b" containerID="cri-o://24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7" gracePeriod=2
Oct 31 01:00:33.176063 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.176037    4478 scope.go:122] "RemoveContainer" containerID="f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00"
Oct 31 01:00:33.176431 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:33.176410    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00\": container with ID starting with f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00 not found: ID does not exist" containerID="f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00"
Oct 31 01:00:33.176527 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.176443    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00"} err="failed to get container status \"f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00\": rpc error: code = NotFound desc = could not find container \"f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00\": container with ID starting with f53ddd578c9d3e52a949d6e322e10f69511526b9b5cf8ae520a6d6d09b6fef00 not found: ID does not exist"
Oct 31 01:00:33.353108 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.353068    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:00:33.353562 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:33.353144    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:00:34.728074 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.728036    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"]
Oct 31 01:00:34.728670 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.728654    4478 cpu_manager.go:414] "containerMap: removing container" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" containerName="test-curl-7498-13087b"
Oct 31 01:00:34.728722 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.728671    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" containerName="test-curl-7498-13087b"
Oct 31 01:00:34.728797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.728787    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" containerName="test-curl-7498-13087b"
Oct 31 01:00:34.733438 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.733420    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:34.792245 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.792217    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"]
Oct 31 01:00:34.837620 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.837591    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-e8808d-dockercfg-r7xm6\""
Oct 31 01:00:34.862368 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.862342    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-e8808d\""
Oct 31 01:00:34.905402 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.902052    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:34.905402 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:34.902192    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbcvx\" (UniqueName: \"kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx\") pod \"payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:35.003527 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.003434    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbcvx\" (UniqueName: \"kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx\") pod \"payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:35.003772 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.003623    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:35.004373 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.004352    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:35.013710 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.013684    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"]
Oct 31 01:00:35.017598 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.017580    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.079410 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.070214    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-aaa842-openshift-default-dockercfg-pg2wn\""
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.102776    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"]
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105085    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105135    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105229    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105291    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105345    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105444    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xh9wz\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105477    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105514    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.105542    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.135407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.116425    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbcvx\" (UniqueName: \"kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx\") pod \"payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:35.212749 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.208277    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.212749 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.208361    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xh9wz\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.221465 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.219122    4478 patch_prober.go:29] interesting pod/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:00:35.221465 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.219233    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:00:35.223455 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223415    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.223618 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223472    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.223618 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223504    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.223618 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223568    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.223618 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223609    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.223841 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223720    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.223841 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.223798    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.224684 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.224658    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.224963 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.224940    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.225599 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.225559    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.225815 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.225781    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.226301 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.226258    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.228402 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.228335    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.240053 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.235680    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.288458 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.287465    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xh9wz\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.290794 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.290741    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token\") pod \"e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") " pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.331228 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.331195    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:35.346287 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.346264    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:35.548594 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.548534    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:35.629327 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.629290    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g\" (UniqueName: \"kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g\") pod \"da736741-54b9-404b-906f-7f779cd3a9bd\" (UID: \"da736741-54b9-404b-906f-7f779cd3a9bd\") "
Oct 31 01:00:35.632246 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.632211    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g" pod "da736741-54b9-404b-906f-7f779cd3a9bd" (UID: "da736741-54b9-404b-906f-7f779cd3a9bd"). InnerVolumeSpecName "kube-api-access-vms7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:35.731290 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.731260    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vms7g\" (UniqueName: \"kubernetes.io/projected/da736741-54b9-404b-906f-7f779cd3a9bd-kube-api-access-vms7g\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:35.791992 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.791960    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"]
Oct 31 01:00:35.792302 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:00:35.792279    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8956368a_0204_460d_be60_2d001593ad91.slice/crio-133df148bc81584b747241021ea9237eeec50edb418f3fd86fca31212d1ed013 WatchSource:0}: Error finding container 133df148bc81584b747241021ea9237eeec50edb418f3fd86fca31212d1ed013: Status 404 returned error can't find the container with id 133df148bc81584b747241021ea9237eeec50edb418f3fd86fca31212d1ed013
Oct 31 01:00:35.796950 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.796932    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"]
Oct 31 01:00:35.812717 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:00:35.812692    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcfb8348_bdee_4b32_be7b_955ea83f8707.slice/crio-b9eed088d799dc2eab7281dbe793237ab7dbfcc59e16a77d573fd989e32cd6f2 WatchSource:0}: Error finding container b9eed088d799dc2eab7281dbe793237ab7dbfcc59e16a77d573fd989e32cd6f2: Status 404 returned error can't find the container with id b9eed088d799dc2eab7281dbe793237ab7dbfcc59e16a77d573fd989e32cd6f2
Oct 31 01:00:35.815659 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.815626    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:00:35.815748 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.815711    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:00:35.815793 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:35.815755    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:00:36.268991 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.268946    4478 generic.go:356] "Generic (PLEG): container finished" podID="da736741-54b9-404b-906f-7f779cd3a9bd" containerID="24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7" exitCode=137
Oct 31 01:00:36.269157 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.269043    4478 scope.go:122] "RemoveContainer" containerID="24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7"
Oct 31 01:00:36.269157 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.269084    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:36.271314 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.271278    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" event={"ID":"bcfb8348-bdee-4b32-be7b-955ea83f8707","Type":"ContainerStarted","Data":"6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"}
Oct 31 01:00:36.271486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.271353    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" event={"ID":"bcfb8348-bdee-4b32-be7b-955ea83f8707","Type":"ContainerStarted","Data":"b9eed088d799dc2eab7281dbe793237ab7dbfcc59e16a77d573fd989e32cd6f2"}
Oct 31 01:00:36.273419 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.273340    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2" event={"ID":"8956368a-0204-460d-be60-2d001593ad91","Type":"ContainerStarted","Data":"f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08"}
Oct 31 01:00:36.273419 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.273420    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2" event={"ID":"8956368a-0204-460d-be60-2d001593ad91","Type":"ContainerStarted","Data":"133df148bc81584b747241021ea9237eeec50edb418f3fd86fca31212d1ed013"}
Oct 31 01:00:36.273629 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.273562    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:36.282328 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.282300    4478 scope.go:122] "RemoveContainer" containerID="24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7"
Oct 31 01:00:36.282835 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:36.282801    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7\": container with ID starting with 24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7 not found: ID does not exist" containerID="24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7"
Oct 31 01:00:36.282923 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.282844    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7"} err="failed to get container status \"24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7\": rpc error: code = NotFound desc = could not find container \"24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7\": container with ID starting with 24837ba3cd77e89277a0cfb5754ff37ed96e574e4db743338a630ca10fd4cdd7 not found: ID does not exist"
Oct 31 01:00:36.331563 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.331535    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:36.337536 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.337512    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:36.392327 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.392240    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podStartSLOduration=2.392226269 podStartE2EDuration="2.392226269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:36.390860587 +0000 UTC m=+3469.407525312" watchObservedRunningTime="2026-10-31 01:00:36.392226269 +0000 UTC m=+3469.408890981"
Oct 31 01:00:36.582601 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:36.582566    4478 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-13087b\" is forbidden: User \"system:node:ip-10-0-137-148.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.us-west-1.compute.internal' and this object" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:36.701256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:36.701201    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2" podStartSLOduration=2.701185823 podStartE2EDuration="2.701185823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:36.697687749 +0000 UTC m=+3469.714352460" watchObservedRunningTime="2026-10-31 01:00:36.701185823 +0000 UTC m=+3469.717850535"
Oct 31 01:00:37.120265 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.119977    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" path="/var/lib/kubelet/pods/da736741-54b9-404b-906f-7f779cd3a9bd/volumes"
Oct 31 01:00:37.165081 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:37.165016    4478 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7498-13087b\" is forbidden: User \"system:node:ip-10-0-137-148.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.us-west-1.compute.internal' and this object" podUID="da736741-54b9-404b-906f-7f779cd3a9bd" pod="opendatahub/test-curl-7498-13087b"
Oct 31 01:00:37.244518 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.244462    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 01:00:37.253207 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253179    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253336 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253243    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253336 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253307    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253461 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253345    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253461 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253448    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253578 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253498    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253578 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253565    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.253679 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253621    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.253730 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253668    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:37.253730 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.253684    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.254048 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.254024    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.254135 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.254093    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.254193 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.254176    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.254193 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.254190    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.254310 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.254200    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.254310 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.254210    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.256438 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.256416    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.256551 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.256434    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:37.256956 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.256934    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "kube-api-access-q9bdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:37.258069 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.258046    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:00:37.278505 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.278482    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 01:00:37.280313 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.280290    4478 generic.go:356] "Generic (PLEG): container finished" podID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerID="5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055" exitCode=0
Oct 31 01:00:37.280424 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.280339    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"
Oct 31 01:00:37.280424 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.280371    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" event={"ID":"86ad1b60-b6f2-485b-bcc1-a95f347faa6e","Type":"ContainerDied","Data":"5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055"}
Oct 31 01:00:37.280539 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.280445    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b" event={"ID":"86ad1b60-b6f2-485b-bcc1-a95f347faa6e","Type":"ContainerDied","Data":"326cdcf0f0e9861671cfd902f04ad87ea0ac57641270e4b12fcf96e79ff944c1"}
Oct 31 01:00:37.280539 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.280469    4478 scope.go:122] "RemoveContainer" containerID="5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055"
Oct 31 01:00:37.281984 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.281963    4478 generic.go:356] "Generic (PLEG): container finished" podID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerID="f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1" exitCode=0
Oct 31 01:00:37.282076 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.282011    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"
Oct 31 01:00:37.282076 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.282051    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" event={"ID":"5480a159-4564-4f5a-9f63-23aa9c6f0708","Type":"ContainerDied","Data":"f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1"}
Oct 31 01:00:37.282191 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.282096    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49" event={"ID":"5480a159-4564-4f5a-9f63-23aa9c6f0708","Type":"ContainerDied","Data":"0b85ab0db4c385d331c96cfe0253600dbfa6ada94d5ce6321e17cbc72d1690e4"}
Oct 31 01:00:37.283574 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.283539    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:37.284393 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.284363    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:00:37.290457 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.290438    4478 scope.go:122] "RemoveContainer" containerID="5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055"
Oct 31 01:00:37.290778 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:37.290759    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055\": container with ID starting with 5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055 not found: ID does not exist" containerID="5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055"
Oct 31 01:00:37.290845 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.290785    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055"} err="failed to get container status \"5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055\": rpc error: code = NotFound desc = could not find container \"5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055\": container with ID starting with 5e03bf0165e0bd09e82face8e4ce520a484ea6b1c105eb2be1205e393fb20055 not found: ID does not exist"
Oct 31 01:00:37.290845 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.290803    4478 scope.go:122] "RemoveContainer" containerID="f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1"
Oct 31 01:00:37.299510 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.299492    4478 scope.go:122] "RemoveContainer" containerID="f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1"
Oct 31 01:00:37.299781 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:37.299762    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1\": container with ID starting with f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1 not found: ID does not exist" containerID="f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1"
Oct 31 01:00:37.299846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.299786    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1"} err="failed to get container status \"f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1\": rpc error: code = NotFound desc = could not find container \"f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1\": container with ID starting with f2cd7fe803e8b6d5f0d27ed7f6260d2790873ba0288766a127b3a926d31bf0d1 not found: ID does not exist"
Oct 31 01:00:37.355177 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355141    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy\") pod \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\" (UID: \"86ad1b60-b6f2-485b-bcc1-a95f347faa6e\") "
Oct 31 01:00:37.355305 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355198    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355305 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355232    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355305 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355269    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355482 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355315    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355482 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355351    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355482 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355408    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355636 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355490    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355636 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355605    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355636 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355611    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.355783 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355674    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.355783 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355767    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket\") pod \"5480a159-4564-4f5a-9f63-23aa9c6f0708\" (UID: \"5480a159-4564-4f5a-9f63-23aa9c6f0708\") "
Oct 31 01:00:37.355973 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.355951    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356410    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356446    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9bdg\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-kube-api-access-q9bdg\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356462    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5480a159-4564-4f5a-9f63-23aa9c6f0708-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356477    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356493    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356509    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356524    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356416    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.356918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.356862    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.358159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.358041    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "kube-api-access-wb7z5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:37.358159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.358119    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:00:37.358354 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.358333    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy" pod "86ad1b60-b6f2-485b-bcc1-a95f347faa6e" (UID: "86ad1b60-b6f2-485b-bcc1-a95f347faa6e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.358427 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.358405    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:00:37.358595 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.358578    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy" pod "5480a159-4564-4f5a-9f63-23aa9c6f0708" (UID: "5480a159-4564-4f5a-9f63-23aa9c6f0708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:00:37.457145 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457107    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.457145 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457136    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86ad1b60-b6f2-485b-bcc1-a95f347faa6e-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.457145 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457146    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.457486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457158    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb7z5\" (UniqueName: \"kubernetes.io/projected/5480a159-4564-4f5a-9f63-23aa9c6f0708-kube-api-access-wb7z5\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.457486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457167    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.457486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457175    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.457486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.457183    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5480a159-4564-4f5a-9f63-23aa9c6f0708-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:00:37.776928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.776893    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"]
Oct 31 01:00:37.783457 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.783431    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-2nv49"]
Oct 31 01:00:37.895405 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.895294    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"]
Oct 31 01:00:37.897859 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:37.897835    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e1ca71-openshift-default-74d7cc5765-mjh7b"]
Oct 31 01:00:39.117792 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:39.117759    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" path="/var/lib/kubelet/pods/5480a159-4564-4f5a-9f63-23aa9c6f0708/volumes"
Oct 31 01:00:39.118213 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:39.118199    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" path="/var/lib/kubelet/pods/86ad1b60-b6f2-485b-bcc1-a95f347faa6e/volumes"
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.991592    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"]
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.992533    4478 cpu_manager.go:414] "containerMap: removing container" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy"
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.992553    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy"
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.992605    4478 cpu_manager.go:414] "containerMap: removing container" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy"
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.992615    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy"
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.992773    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="86ad1b60-b6f2-485b-bcc1-a95f347faa6e" containerName="istio-proxy"
Oct 31 01:00:42.994413 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:42.992792    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="5480a159-4564-4f5a-9f63-23aa9c6f0708" containerName="istio-proxy"
Oct 31 01:00:43.010414 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.009954    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.053927 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.053817    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"]
Oct 31 01:00:43.093646 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.093613    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-aaa842\""
Oct 31 01:00:43.093814 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.093652    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-aaa842-dockercfg-qzvvh\""
Oct 31 01:00:43.122165 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.122138    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.122317 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.122177    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d29hx\" (UniqueName: \"kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx\") pod \"payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.223329 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.223294    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.223548 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.223346    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d29hx\" (UniqueName: \"kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx\") pod \"payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.224067 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.224044    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.287452 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.285587    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d29hx\" (UniqueName: \"kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx\") pod \"payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.356583 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.356546    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:43.755974 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:43.755943    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"]
Oct 31 01:00:43.756688 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:00:43.756655    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb188ed46_edd2_4d5c_a08e_e744043e7f99.slice/crio-2f0efc7f07b45ebaf66068b3edae4951ff76047b3bfc0c6f3bc9cac4272a31e2 WatchSource:0}: Error finding container 2f0efc7f07b45ebaf66068b3edae4951ff76047b3bfc0c6f3bc9cac4272a31e2: Status 404 returned error can't find the container with id 2f0efc7f07b45ebaf66068b3edae4951ff76047b3bfc0c6f3bc9cac4272a31e2
Oct 31 01:00:44.329264 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:44.329221    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs" event={"ID":"b188ed46-edd2-4d5c-a08e-e744043e7f99","Type":"ContainerStarted","Data":"fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a"}
Oct 31 01:00:44.329773 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:44.329404    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:00:44.329773 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:44.329431    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs" event={"ID":"b188ed46-edd2-4d5c-a08e-e744043e7f99","Type":"ContainerStarted","Data":"2f0efc7f07b45ebaf66068b3edae4951ff76047b3bfc0c6f3bc9cac4272a31e2"}
Oct 31 01:00:44.446626 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:44.446567    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs" podStartSLOduration=2.446549767 podStartE2EDuration="2.446549767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:00:44.443915003 +0000 UTC m=+3477.460579727" watchObservedRunningTime="2026-10-31 01:00:44.446549767 +0000 UTC m=+3477.463214480"
Oct 31 01:00:44.787438 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:00:44.786643    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:00:47.286515 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:47.286488    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:00:55.339075 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:00:55.339049    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:01:14.771650 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:01:14.771551    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:01:17.741896 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:17.741862    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7504-7e2255"]
Oct 31 01:01:17.746108 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:17.746081    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:17.801936 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:17.801907    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-7e2255"]
Oct 31 01:01:17.839389 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:17.839361    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8n9ml\" (UniqueName: \"kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml\") pod \"test-curl-7504-7e2255\" (UID: \"977b99fd-eaac-4baa-8759-777dc13c0550\") " pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:17.847175 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:17.847148    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-9d9vc\""
Oct 31 01:01:17.940465 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:17.940423    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8n9ml\" (UniqueName: \"kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml\") pod \"test-curl-7504-7e2255\" (UID: \"977b99fd-eaac-4baa-8759-777dc13c0550\") " pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:18.001466 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:18.001393    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8n9ml\" (UniqueName: \"kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml\") pod \"test-curl-7504-7e2255\" (UID: \"977b99fd-eaac-4baa-8759-777dc13c0550\") " pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:18.058124 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:18.058089    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:18.439907 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:18.439877    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7504-7e2255"]
Oct 31 01:01:18.442021 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:01:18.441993    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977b99fd_eaac_4baa_8759_777dc13c0550.slice/crio-91058941838cab2cc28c49c84df3dfa6ae5f100af29f8c4bb15052f60ee4a55f WatchSource:0}: Error finding container 91058941838cab2cc28c49c84df3dfa6ae5f100af29f8c4bb15052f60ee4a55f: Status 404 returned error can't find the container with id 91058941838cab2cc28c49c84df3dfa6ae5f100af29f8c4bb15052f60ee4a55f
Oct 31 01:01:18.547618 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:18.547583    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-7e2255" event={"ID":"977b99fd-eaac-4baa-8759-777dc13c0550","Type":"ContainerStarted","Data":"39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e"}
Oct 31 01:01:18.547785 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:18.547632    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7504-7e2255" event={"ID":"977b99fd-eaac-4baa-8759-777dc13c0550","Type":"ContainerStarted","Data":"91058941838cab2cc28c49c84df3dfa6ae5f100af29f8c4bb15052f60ee4a55f"}
Oct 31 01:01:18.661364 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:18.661313    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7504-7e2255" podStartSLOduration=1.661298474 podStartE2EDuration="1.661298474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:18.659854144 +0000 UTC m=+3511.676518878" watchObservedRunningTime="2026-10-31 01:01:18.661298474 +0000 UTC m=+3511.677963187"
Oct 31 01:01:20.169231 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:20.169196    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7504-7e2255"]
Oct 31 01:01:20.174496 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:20.174462    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7504-7e2255"]
Oct 31 01:01:20.558999 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:20.558956    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7504-7e2255" podUID="977b99fd-eaac-4baa-8759-777dc13c0550" containerName="test-curl-7504-7e2255" containerID="cri-o://39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e" gracePeriod=2
Oct 31 01:01:20.609813 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:01:20.609767    4478 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7504-7e2255\" is forbidden: User \"system:node:ip-10-0-137-148.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-137-148.us-west-1.compute.internal' and this object" podUID="977b99fd-eaac-4baa-8759-777dc13c0550" pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:22.808701 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:22.808677    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:22.896945 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:22.896870    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml\" (UniqueName: \"kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml\") pod \"977b99fd-eaac-4baa-8759-777dc13c0550\" (UID: \"977b99fd-eaac-4baa-8759-777dc13c0550\") "
Oct 31 01:01:22.898984 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:22.898959    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml" pod "977b99fd-eaac-4baa-8759-777dc13c0550" (UID: "977b99fd-eaac-4baa-8759-777dc13c0550"). InnerVolumeSpecName "kube-api-access-8n9ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:01:22.998712 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:22.998683    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8n9ml\" (UniqueName: \"kubernetes.io/projected/977b99fd-eaac-4baa-8759-777dc13c0550-kube-api-access-8n9ml\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:01:23.117464 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:23.117425    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977b99fd-eaac-4baa-8759-777dc13c0550" path="/var/lib/kubelet/pods/977b99fd-eaac-4baa-8759-777dc13c0550/volumes"
Oct 31 01:01:23.577859 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:23.577823    4478 generic.go:356] "Generic (PLEG): container finished" podID="977b99fd-eaac-4baa-8759-777dc13c0550" containerID="39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e" exitCode=137
Oct 31 01:01:23.578037 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:23.577873    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7504-7e2255"
Oct 31 01:01:23.578037 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:23.577908    4478 scope.go:122] "RemoveContainer" containerID="39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e"
Oct 31 01:01:23.587153 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:23.587131    4478 scope.go:122] "RemoveContainer" containerID="39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e"
Oct 31 01:01:23.587485 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:01:23.587467    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e\": container with ID starting with 39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e not found: ID does not exist" containerID="39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e"
Oct 31 01:01:23.587555 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:23.587499    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e"} err="failed to get container status \"39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e\": rpc error: code = NotFound desc = could not find container \"39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e\": container with ID starting with 39d81f3511fe070279f35dccbcfa4c573634fef0c2daa39edb78bf4c0d723b7e not found: ID does not exist"
Oct 31 01:01:43.410452 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.410413    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"]
Oct 31 01:01:43.411005 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.410991    4478 cpu_manager.go:414] "containerMap: removing container" podUID="977b99fd-eaac-4baa-8759-777dc13c0550" containerName="test-curl-7504-7e2255"
Oct 31 01:01:43.411049 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.411007    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977b99fd-eaac-4baa-8759-777dc13c0550" containerName="test-curl-7504-7e2255"
Oct 31 01:01:43.411116 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.411106    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="977b99fd-eaac-4baa-8759-777dc13c0550" containerName="test-curl-7504-7e2255"
Oct 31 01:01:43.415779 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.415762    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.470795 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.470765    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-e8808d-serving-cert\""
Oct 31 01:01:43.517027 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.516993    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.517204 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.517146    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.517265 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.517211    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2w52\" (UniqueName: \"kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.517340 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.517317    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.523103 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.523076    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"]
Oct 31 01:01:43.618782 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.618744    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.618970 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.618857    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2w52\" (UniqueName: \"kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.618970 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.618921    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.619086 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.619068    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.619146 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:01:43.619095    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-serving-cert: secret "maas-api-e2e-shared-b-w1-e8808d-serving-cert" not found
Oct 31 01:01:43.619197 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:01:43.619157    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls podName:a06dc4e0-6c1b-454c-b86d-84c87bcdf882 nodeName:}" failed. No retries permitted until 2026-10-31 01:01:44.1191385 +0000 UTC m=+3537.135803203 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls") pod "maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" (UID: "a06dc4e0-6c1b-454c-b86d-84c87bcdf882") : secret "maas-api-e2e-shared-b-w1-e8808d-serving-cert" not found
Oct 31 01:01:43.619523 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.619500    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.621633 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.621616    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:43.694011 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:43.693952    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2w52\" (UniqueName: \"kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:44.135592 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:44.135507    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:44.138166 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:44.138135    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:44.329611 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:44.329581    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:44.711428 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:44.711403    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"]
Oct 31 01:01:44.714215 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:01:44.714180    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06dc4e0_6c1b_454c_b86d_84c87bcdf882.slice/crio-a808ae98d7ec9864da0133d00f5a4279645878134cf9bf3f1629ef893e91ce0e WatchSource:0}: Error finding container a808ae98d7ec9864da0133d00f5a4279645878134cf9bf3f1629ef893e91ce0e: Status 404 returned error can't find the container with id a808ae98d7ec9864da0133d00f5a4279645878134cf9bf3f1629ef893e91ce0e
Oct 31 01:01:44.757719 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:01:44.757685    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:01:45.721357 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:45.721325    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" event={"ID":"a06dc4e0-6c1b-454c-b86d-84c87bcdf882","Type":"ContainerStarted","Data":"a808ae98d7ec9864da0133d00f5a4279645878134cf9bf3f1629ef893e91ce0e"}
Oct 31 01:01:46.728953 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:46.728920    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" event={"ID":"a06dc4e0-6c1b-454c-b86d-84c87bcdf882","Type":"ContainerStarted","Data":"475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02"}
Oct 31 01:01:46.729566 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:46.729098    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:46.848053 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:46.847999    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" podStartSLOduration=2.856163289 podStartE2EDuration="3.847983696s" totalImagesPullingTime="991.820407ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:46.845453035 +0000 UTC m=+3539.862117748" watchObservedRunningTime="2026-10-31 01:01:46.847983696 +0000 UTC m=+3539.864648453"
Oct 31 01:01:52.743029 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:52.743001    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:01:57.905776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:57.905730    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"]
Oct 31 01:01:57.909710 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:57.909692    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:57.969046 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:57.969017    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"]
Oct 31 01:01:58.013624 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.013593    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-cc6150\""
Oct 31 01:01:58.013947 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.013930    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-cc6150-dockercfg-lg5tr\""
Oct 31 01:01:58.022730 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.022693    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjv8\" (UniqueName: \"kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8\") pod \"payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.023075 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.023054    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.059898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.059855    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"]
Oct 31 01:01:58.065092 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.065073    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.122501 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.122462    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"]
Oct 31 01:01:58.124833 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.124805    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.124998 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.124929    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjv8\" (UniqueName: \"kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8\") pod \"payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.125916 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.125876    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.185789 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.185764    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjv8\" (UniqueName: \"kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8\") pod \"payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.223653 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.223625    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.226100 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.226076    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.226208 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.226185    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xtk\" (UniqueName: \"kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk\") pod \"payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.327774 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.327728    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.327974 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.327849    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xtk\" (UniqueName: \"kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk\") pod \"payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.328751 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.328730    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.399973 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.399916    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xtk\" (UniqueName: \"kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk\") pod \"payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.614660 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.614634    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"]
Oct 31 01:01:58.616781 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:01:58.616750    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0956f55c_2fc4_4f39_baed_0232d37f4265.slice/crio-2007ef05c8701df6cd2103f3bc5e74fcde1ce18cce16401851944471a51819d0 WatchSource:0}: Error finding container 2007ef05c8701df6cd2103f3bc5e74fcde1ce18cce16401851944471a51819d0: Status 404 returned error can't find the container with id 2007ef05c8701df6cd2103f3bc5e74fcde1ce18cce16401851944471a51819d0
Oct 31 01:01:58.682324 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.682299    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:58.847055 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.847011    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898" event={"ID":"0956f55c-2fc4-4f39-baed-0232d37f4265","Type":"ContainerStarted","Data":"82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733"}
Oct 31 01:01:58.847242 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.847087    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898" event={"ID":"0956f55c-2fc4-4f39-baed-0232d37f4265","Type":"ContainerStarted","Data":"2007ef05c8701df6cd2103f3bc5e74fcde1ce18cce16401851944471a51819d0"}
Oct 31 01:01:58.847321 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.847305    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:01:58.963604 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:58.963552    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898" podStartSLOduration=1.963537334 podStartE2EDuration="1.963537334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:58.961818177 +0000 UTC m=+3551.978482922" watchObservedRunningTime="2026-10-31 01:01:58.963537334 +0000 UTC m=+3551.980202045"
Oct 31 01:01:59.066783 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:59.066749    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"]
Oct 31 01:01:59.068994 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:01:59.068964    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22810094_24e8_4b47_9b86_b70056819c27.slice/crio-f7e2135185d650c7586b714f492d999de7de5bad9838f5146e9291654c164331 WatchSource:0}: Error finding container f7e2135185d650c7586b714f492d999de7de5bad9838f5146e9291654c164331: Status 404 returned error can't find the container with id f7e2135185d650c7586b714f492d999de7de5bad9838f5146e9291654c164331
Oct 31 01:01:59.855694 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:59.855659    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f" event={"ID":"22810094-24e8-4b47-9b86-b70056819c27","Type":"ContainerStarted","Data":"024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387"}
Oct 31 01:01:59.855914 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:59.855717    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f" event={"ID":"22810094-24e8-4b47-9b86-b70056819c27","Type":"ContainerStarted","Data":"f7e2135185d650c7586b714f492d999de7de5bad9838f5146e9291654c164331"}
Oct 31 01:01:59.855914 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:59.855764    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:01:59.974138 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:01:59.974081    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f" podStartSLOduration=1.974063289 podStartE2EDuration="1.974063289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:01:59.970595197 +0000 UTC m=+3552.987259908" watchObservedRunningTime="2026-10-31 01:01:59.974063289 +0000 UTC m=+3552.990728003"
Oct 31 01:02:09.860275 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:09.860239    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:02:10.866642 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:10.866601    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:02:14.760361 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:02:14.760323    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:02:43.217014 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:43.216978    4478 generic.go:356] "Generic (PLEG): container finished" podID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerID="6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e" exitCode=0
Oct 31 01:02:43.217446 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:43.217054    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" event={"ID":"bcfb8348-bdee-4b32-be7b-955ea83f8707","Type":"ContainerDied","Data":"6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"}
Oct 31 01:02:43.217495 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:43.217460    4478 scope.go:122] "RemoveContainer" containerID="6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"
Oct 31 01:02:43.218996 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:43.218970    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:02:43.219063 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:43.219039    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:02:43.219120 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:43.219106    4478 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Oct 31 01:02:44.225760 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:44.225721    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" event={"ID":"bcfb8348-bdee-4b32-be7b-955ea83f8707","Type":"ContainerStarted","Data":"bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328"}
Oct 31 01:02:44.340406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:44.334689    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:02:44.340406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:44.334808    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:02:44.340406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:44.334850    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 01:02:44.785140 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:02:44.785104    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:02:45.371849 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:45.370615    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:02:45.371849 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:45.370674    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 01:02:46.333005 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:46.332967    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:02:46.333282 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:46.333251    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 01:02:47.335776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:47.335741    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:02:47.336407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:47.336356    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 01:02:48.352407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:48.351493    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:02:48.352407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:48.351557    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 01:02:49.331829 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:49.331788    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused" start-of-body=
Oct 31 01:02:49.332018 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:49.331875    4478 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.152:15021/healthz/ready\": dial tcp 10.132.0.152:15021: connect: connection refused"
Oct 31 01:02:50.336429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:50.336373    4478 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:02:50.337703 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:50.336869    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:02:50.337703 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:02:50.337626    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:03:14.759122 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:03:14.759085    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:03:28.618311 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.618236    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"]
Oct 31 01:03:28.622306 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.622290    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.683069 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.683042    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"]
Oct 31 01:03:28.684983 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.684954    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.685220 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.685194    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4m8\" (UniqueName: \"kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8\") pod \"payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.730812 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.730781    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-fae60c-dockercfg-mwlrl\""
Oct 31 01:03:28.767406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.735327    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-fae60c\""
Oct 31 01:03:28.787144 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.787100    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4m8\" (UniqueName: \"kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8\") pod \"payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.794655 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.794604    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.796065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.795794    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.849374 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.849339    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4m8\" (UniqueName: \"kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8\") pod \"payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:28.935943 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:28.935911    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:29.317118 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:29.317043    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"]
Oct 31 01:03:29.320893 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:03:29.320864    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda55133d3_f8dd_436a_93f4_d6eeca18b4f0.slice/crio-388ddaf45c2255912e76f9c6c26dcb5979961f51d3a405a36abb014e786ad929 WatchSource:0}: Error finding container 388ddaf45c2255912e76f9c6c26dcb5979961f51d3a405a36abb014e786ad929: Status 404 returned error can't find the container with id 388ddaf45c2255912e76f9c6c26dcb5979961f51d3a405a36abb014e786ad929
Oct 31 01:03:29.565674 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:29.565621    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv" event={"ID":"a55133d3-f8dd-436a-93f4-d6eeca18b4f0","Type":"ContainerStarted","Data":"e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558"}
Oct 31 01:03:29.565852 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:29.565693    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv" event={"ID":"a55133d3-f8dd-436a-93f4-d6eeca18b4f0","Type":"ContainerStarted","Data":"388ddaf45c2255912e76f9c6c26dcb5979961f51d3a405a36abb014e786ad929"}
Oct 31 01:03:29.565852 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:29.565721    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:29.681219 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:29.681173    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv" podStartSLOduration=1.6811558739999999 podStartE2EDuration="1.681155874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:29.680229932 +0000 UTC m=+3642.696894665" watchObservedRunningTime="2026-10-31 01:03:29.681155874 +0000 UTC m=+3642.697820589"
Oct 31 01:03:40.575585 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:40.575558    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:03:44.797657 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:03:44.797561    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:03:44.880119 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:44.880092    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"]
Oct 31 01:03:44.884397 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:44.884365    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:44.939321 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:44.939293    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-395b06\""
Oct 31 01:03:44.989772 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:44.989736    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"]
Oct 31 01:03:45.000135 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.000102    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.000271 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.000150    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm5wg\" (UniqueName: \"kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg\") pod \"payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.009478 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.009445    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-395b06-dockercfg-lxlpc\""
Oct 31 01:03:45.101503 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.101413    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.101503 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.101477    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm5wg\" (UniqueName: \"kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg\") pod \"payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.102070 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.102051    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.162829 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.162772    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm5wg\" (UniqueName: \"kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg\") pod \"payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.200261 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.200230    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.590682 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.590654    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"]
Oct 31 01:03:45.592309 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:03:45.592280    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd11ca345_48df_4668_9134_a3836787ba57.slice/crio-438f86e1ad3ebe5d0e614bb747260d295255194e771ed4656914f8532be3699c WatchSource:0}: Error finding container 438f86e1ad3ebe5d0e614bb747260d295255194e771ed4656914f8532be3699c: Status 404 returned error can't find the container with id 438f86e1ad3ebe5d0e614bb747260d295255194e771ed4656914f8532be3699c
Oct 31 01:03:45.671514 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.671485    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn" event={"ID":"d11ca345-48df-4668-9134-a3836787ba57","Type":"ContainerStarted","Data":"26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5"}
Oct 31 01:03:45.671710 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.671540    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn" event={"ID":"d11ca345-48df-4668-9134-a3836787ba57","Type":"ContainerStarted","Data":"438f86e1ad3ebe5d0e614bb747260d295255194e771ed4656914f8532be3699c"}
Oct 31 01:03:45.671781 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.671706    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:03:45.797228 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:45.797179    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn" podStartSLOduration=1.797165882 podStartE2EDuration="1.797165882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:03:45.793532808 +0000 UTC m=+3658.810197520" watchObservedRunningTime="2026-10-31 01:03:45.797165882 +0000 UTC m=+3658.813830595"
Oct 31 01:03:56.681244 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:03:56.681209    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:04:14.759504 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:04:14.759468    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:04:21.891687 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.891655    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"]
Oct 31 01:04:21.894944 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.894927    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:21.946946 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.946910    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:21.947101 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.947045    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:21.947170 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.947132    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w24b\" (UniqueName: \"kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:21.947234 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.947195    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:21.947290 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:21.947255    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-2aebbd-serving-cert\""
Oct 31 01:04:22.000544 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.000516    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"]
Oct 31 01:04:22.048178 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.048144    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.048346 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.048219    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.048346 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.048334    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w24b\" (UniqueName: \"kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.048498 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.048412    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.049406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.049364    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.049538 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:04:22.049491    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-serving-cert: secret "maas-api-e2e-extmod-a-2aebbd-serving-cert" not found
Oct 31 01:04:22.049601 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:04:22.049562    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls podName:033bc0df-a025-4649-8042-0644a89a9102 nodeName:}" failed. No retries permitted until 2026-10-31 01:04:22.549542225 +0000 UTC m=+3695.566206929 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls") pod "maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" (UID: "033bc0df-a025-4649-8042-0644a89a9102") : secret "maas-api-e2e-extmod-a-2aebbd-serving-cert" not found
Oct 31 01:04:22.063924 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.063897    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.110768 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.110732    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w24b\" (UniqueName: \"kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.562841 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.562803    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.565543 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.565513    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\") pod \"maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:22.807543 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:22.807515    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:23.191019 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:23.190994    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"]
Oct 31 01:04:23.192972 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:04:23.192944    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod033bc0df_a025_4649_8042_0644a89a9102.slice/crio-b52331b8f6777c8caeba873ae9c924d89d865f5a8ad148ae65e4cc0f2f7412d5 WatchSource:0}: Error finding container b52331b8f6777c8caeba873ae9c924d89d865f5a8ad148ae65e4cc0f2f7412d5: Status 404 returned error can't find the container with id b52331b8f6777c8caeba873ae9c924d89d865f5a8ad148ae65e4cc0f2f7412d5
Oct 31 01:04:23.929263 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:23.929229    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" event={"ID":"033bc0df-a025-4649-8042-0644a89a9102","Type":"ContainerStarted","Data":"b52331b8f6777c8caeba873ae9c924d89d865f5a8ad148ae65e4cc0f2f7412d5"}
Oct 31 01:04:24.973101 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:24.973062    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" event={"ID":"033bc0df-a025-4649-8042-0644a89a9102","Type":"ContainerStarted","Data":"933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321"}
Oct 31 01:04:24.974155 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:24.974131    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:25.094627 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:25.094568    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" podStartSLOduration=3.268986558 podStartE2EDuration="4.094553341s" totalImagesPullingTime="825.566783ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:04:25.090746977 +0000 UTC m=+3698.107411741" watchObservedRunningTime="2026-10-31 01:04:25.094553341 +0000 UTC m=+3698.111218052"
Oct 31 01:04:31.994037 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:04:31.994007    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:04:44.757836 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:04:44.757800    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:05:00.177317 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.177242    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"]
Oct 31 01:05:00.181496 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.181481    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.208304 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.208272    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrgs7\" (UniqueName: \"kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.208471 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.208321    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.208553 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.208520    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.208623 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.208570    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.236372 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.236347    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-fae60c-serving-cert\""
Oct 31 01:05:00.343405 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.314590    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrgs7\" (UniqueName: \"kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.343405 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.314644    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.343405 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.314747    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.343405 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.314795    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.343405 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:05:00.314974    4478 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-serving-cert: secret "maas-api-e2e-extmod-a-fae60c-serving-cert" not found
Oct 31 01:05:00.343405 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:05:00.315036    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls podName:ecfea579-6973-4647-9bd4-3edc50217db8 nodeName:}" failed. No retries permitted until 2026-10-31 01:05:00.815016483 +0000 UTC m=+3733.831681195 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls") pod "maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" (UID: "ecfea579-6973-4647-9bd4-3edc50217db8") : secret "maas-api-e2e-extmod-a-fae60c-serving-cert" not found
Oct 31 01:05:00.389633 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.388971    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.389989 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.389964    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.392411 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.392331    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"]
Oct 31 01:05:00.402959 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.395671    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrgs7\" (UniqueName: \"kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.821052 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.821017    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:00.823818 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:00.823779    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\") pod \"maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:01.095391 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:01.095306    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:01.478596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:01.478573    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"]
Oct 31 01:05:01.480314 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:05:01.480276    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecfea579_6973_4647_9bd4_3edc50217db8.slice/crio-5d92f1b8ca1d11e700803bcb12475ce45de1f907a5efb69d02b30de8109369b5 WatchSource:0}: Error finding container 5d92f1b8ca1d11e700803bcb12475ce45de1f907a5efb69d02b30de8109369b5: Status 404 returned error can't find the container with id 5d92f1b8ca1d11e700803bcb12475ce45de1f907a5efb69d02b30de8109369b5
Oct 31 01:05:02.213263 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:02.213227    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" event={"ID":"ecfea579-6973-4647-9bd4-3edc50217db8","Type":"ContainerStarted","Data":"5d92f1b8ca1d11e700803bcb12475ce45de1f907a5efb69d02b30de8109369b5"}
Oct 31 01:05:03.230110 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:03.230073    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" event={"ID":"ecfea579-6973-4647-9bd4-3edc50217db8","Type":"ContainerStarted","Data":"7baff75df611010160a621187918273751960239b441a78422439a95c890fb28"}
Oct 31 01:05:03.230535 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:03.230191    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:03.352628 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:03.352575    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" podStartSLOduration=2.439898858 podStartE2EDuration="3.352561029s" totalImagesPullingTime="912.662171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:05:03.348299168 +0000 UTC m=+3736.364963896" watchObservedRunningTime="2026-10-31 01:05:03.352561029 +0000 UTC m=+3736.369225740"
Oct 31 01:05:03.486149 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:03.486059    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"]
Oct 31 01:05:03.568880 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:03.568846    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"]
Oct 31 01:05:08.487699 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.487663    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f" podUID="22810094-24e8-4b47-9b86-b70056819c27" containerName="payload-processing" containerID="cri-o://024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387" gracePeriod=25
Oct 31 01:05:08.569236 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.569198    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898" podUID="0956f55c-2fc4-4f39-baed-0232d37f4265" containerName="payload-pre-processing" containerID="cri-o://82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733" gracePeriod=25
Oct 31 01:05:08.846651 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.846623    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:05:08.871967 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.871937    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:05:08.920493 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.920463    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume\") pod \"22810094-24e8-4b47-9b86-b70056819c27\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") "
Oct 31 01:05:08.920645 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.920496    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk\" (UniqueName: \"kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk\") pod \"22810094-24e8-4b47-9b86-b70056819c27\" (UID: \"22810094-24e8-4b47-9b86-b70056819c27\") "
Oct 31 01:05:08.920697 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.920684    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume\") pod \"0956f55c-2fc4-4f39-baed-0232d37f4265\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") "
Oct 31 01:05:08.920783 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.920765    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8\" (UniqueName: \"kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8\") pod \"0956f55c-2fc4-4f39-baed-0232d37f4265\" (UID: \"0956f55c-2fc4-4f39-baed-0232d37f4265\") "
Oct 31 01:05:08.921004 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.920851    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume" pod "22810094-24e8-4b47-9b86-b70056819c27" (UID: "22810094-24e8-4b47-9b86-b70056819c27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:08.921110 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.921016    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume" pod "0956f55c-2fc4-4f39-baed-0232d37f4265" (UID: "0956f55c-2fc4-4f39-baed-0232d37f4265"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:05:08.921341 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.921324    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22810094-24e8-4b47-9b86-b70056819c27-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:08.921425 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.921347    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0956f55c-2fc4-4f39-baed-0232d37f4265-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:08.923164 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.923142    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk" pod "22810094-24e8-4b47-9b86-b70056819c27" (UID: "22810094-24e8-4b47-9b86-b70056819c27"). InnerVolumeSpecName "kube-api-access-s6xtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:08.923351 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:08.923337    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8" pod "0956f55c-2fc4-4f39-baed-0232d37f4265" (UID: "0956f55c-2fc4-4f39-baed-0232d37f4265"). InnerVolumeSpecName "kube-api-access-bhjv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:05:09.022951 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.022880    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjv8\" (UniqueName: \"kubernetes.io/projected/0956f55c-2fc4-4f39-baed-0232d37f4265-kube-api-access-bhjv8\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:09.022951 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.022910    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xtk\" (UniqueName: \"kubernetes.io/projected/22810094-24e8-4b47-9b86-b70056819c27-kube-api-access-s6xtk\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:05:09.241469 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.241442    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:05:09.271481 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.271449    4478 generic.go:356] "Generic (PLEG): container finished" podID="22810094-24e8-4b47-9b86-b70056819c27" containerID="024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387" exitCode=0
Oct 31 01:05:09.271610 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.271485    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"
Oct 31 01:05:09.271610 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.271537    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f" event={"ID":"22810094-24e8-4b47-9b86-b70056819c27","Type":"ContainerDied","Data":"024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387"}
Oct 31 01:05:09.271610 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.271595    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f" event={"ID":"22810094-24e8-4b47-9b86-b70056819c27","Type":"ContainerDied","Data":"f7e2135185d650c7586b714f492d999de7de5bad9838f5146e9291654c164331"}
Oct 31 01:05:09.271780 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.271625    4478 scope.go:122] "RemoveContainer" containerID="024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387"
Oct 31 01:05:09.273179 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.273122    4478 generic.go:356] "Generic (PLEG): container finished" podID="0956f55c-2fc4-4f39-baed-0232d37f4265" containerID="82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733" exitCode=0
Oct 31 01:05:09.273296 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.273227    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898" event={"ID":"0956f55c-2fc4-4f39-baed-0232d37f4265","Type":"ContainerDied","Data":"82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733"}
Oct 31 01:05:09.273296 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.273265    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898" event={"ID":"0956f55c-2fc4-4f39-baed-0232d37f4265","Type":"ContainerDied","Data":"2007ef05c8701df6cd2103f3bc5e74fcde1ce18cce16401851944471a51819d0"}
Oct 31 01:05:09.273448 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.273428    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"
Oct 31 01:05:09.283229 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.283209    4478 scope.go:122] "RemoveContainer" containerID="024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387"
Oct 31 01:05:09.283572 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:05:09.283544    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387\": container with ID starting with 024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387 not found: ID does not exist" containerID="024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387"
Oct 31 01:05:09.283663 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.283580    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387"} err="failed to get container status \"024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387\": rpc error: code = NotFound desc = could not find container \"024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387\": container with ID starting with 024c054f04aecd4115f82f90021308bca2fe1fb57e254c55c7aed560898e3387 not found: ID does not exist"
Oct 31 01:05:09.283663 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.283603    4478 scope.go:122] "RemoveContainer" containerID="82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733"
Oct 31 01:05:09.292536 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.292519    4478 scope.go:122] "RemoveContainer" containerID="82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733"
Oct 31 01:05:09.292783 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:05:09.292765    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733\": container with ID starting with 82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733 not found: ID does not exist" containerID="82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733"
Oct 31 01:05:09.292829 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.292790    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733"} err="failed to get container status \"82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733\": rpc error: code = NotFound desc = could not find container \"82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733\": container with ID starting with 82a2735aa3212653cc245dde91849743fa78b74e2eb8ab3e9015437784872733 not found: ID does not exist"
Oct 31 01:05:09.472754 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.472714    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"]
Oct 31 01:05:09.482532 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.482499    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-cc6150-85d84c4867-kgg8f"]
Oct 31 01:05:09.588120 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.588035    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"]
Oct 31 01:05:09.592162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:09.592136    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-cc6150-557d4b9667-dt898"]
Oct 31 01:05:11.118544 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:11.118507    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0956f55c-2fc4-4f39-baed-0232d37f4265" path="/var/lib/kubelet/pods/0956f55c-2fc4-4f39-baed-0232d37f4265/volumes"
Oct 31 01:05:11.119162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:05:11.119125    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22810094-24e8-4b47-9b86-b70056819c27" path="/var/lib/kubelet/pods/22810094-24e8-4b47-9b86-b70056819c27/volumes"
Oct 31 01:05:14.764427 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:05:14.764360    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:05:44.757816 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:05:44.757769    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:06:04.871819 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:04.871782    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"]
Oct 31 01:06:04.872213 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:04.872005    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" podUID="033bc0df-a025-4649-8042-0644a89a9102" containerName="maas-api" containerID="cri-o://933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321" gracePeriod=30
Oct 31 01:06:05.159674 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.159652    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:06:05.241671 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.241643    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca\") pod \"033bc0df-a025-4649-8042-0644a89a9102\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") "
Oct 31 01:06:05.241864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.241852    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b\" (UniqueName: \"kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b\") pod \"033bc0df-a025-4649-8042-0644a89a9102\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") "
Oct 31 01:06:05.241941 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.241873    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs\") pod \"033bc0df-a025-4649-8042-0644a89a9102\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") "
Oct 31 01:06:05.241941 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.241904    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\") pod \"033bc0df-a025-4649-8042-0644a89a9102\" (UID: \"033bc0df-a025-4649-8042-0644a89a9102\") "
Oct 31 01:06:05.242062 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.242034    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca" pod "033bc0df-a025-4649-8042-0644a89a9102" (UID: "033bc0df-a025-4649-8042-0644a89a9102"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:05.242278 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.242261    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/033bc0df-a025-4649-8042-0644a89a9102-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:05.244223 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.244197    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs" pod "033bc0df-a025-4649-8042-0644a89a9102" (UID: "033bc0df-a025-4649-8042-0644a89a9102"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:06:05.244223 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.244217    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls" pod "033bc0df-a025-4649-8042-0644a89a9102" (UID: "033bc0df-a025-4649-8042-0644a89a9102"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:06:05.244470 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.244313    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b" pod "033bc0df-a025-4649-8042-0644a89a9102" (UID: "033bc0df-a025-4649-8042-0644a89a9102"). InnerVolumeSpecName "kube-api-access-6w24b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:05.343356 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.343323    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w24b\" (UniqueName: \"kubernetes.io/projected/033bc0df-a025-4649-8042-0644a89a9102-kube-api-access-6w24b\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:05.343356 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.343356    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:05.343584 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.343371    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/033bc0df-a025-4649-8042-0644a89a9102-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:05.650776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.650737    4478 generic.go:356] "Generic (PLEG): container finished" podID="033bc0df-a025-4649-8042-0644a89a9102" containerID="933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321" exitCode=0
Oct 31 01:06:05.650964 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.650782    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"
Oct 31 01:06:05.650964 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.650784    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" event={"ID":"033bc0df-a025-4649-8042-0644a89a9102","Type":"ContainerDied","Data":"933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321"}
Oct 31 01:06:05.650964 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.650912    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg" event={"ID":"033bc0df-a025-4649-8042-0644a89a9102","Type":"ContainerDied","Data":"b52331b8f6777c8caeba873ae9c924d89d865f5a8ad148ae65e4cc0f2f7412d5"}
Oct 31 01:06:05.650964 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.650943    4478 scope.go:122] "RemoveContainer" containerID="933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321"
Oct 31 01:06:05.666123 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.666098    4478 scope.go:122] "RemoveContainer" containerID="933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321"
Oct 31 01:06:05.666493 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:05.666468    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321\": container with ID starting with 933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321 not found: ID does not exist" containerID="933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321"
Oct 31 01:06:05.666584 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.666503    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321"} err="failed to get container status \"933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321\": rpc error: code = NotFound desc = could not find container \"933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321\": container with ID starting with 933c65cdc55b2f30d7a5a1d32b2e3943c4debe1c5363665760b136713a63d321 not found: ID does not exist"
Oct 31 01:06:05.815603 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.815570    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"]
Oct 31 01:06:05.864951 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:05.864920    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-2aebbd-7b8f758b49-nkkhg"]
Oct 31 01:06:07.127445 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:07.127403    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="033bc0df-a025-4649-8042-0644a89a9102" path="/var/lib/kubelet/pods/033bc0df-a025-4649-8042-0644a89a9102/volumes"
Oct 31 01:06:14.758316 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:14.758278    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:06:27.994825 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:27.994788    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"]
Oct 31 01:06:28.107206 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:28.107169    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"]
Oct 31 01:06:32.995437 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:32.995396    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" containerName="payload-processing" containerID="cri-o://ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892" gracePeriod=25
Oct 31 01:06:33.107810 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:33.107770    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerName="payload-pre-processing" containerID="cri-o://d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82" gracePeriod=25
Oct 31 01:06:33.917613 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:33.917584    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"]
Oct 31 01:06:37.530223 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.530187    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"]
Oct 31 01:06:37.530677 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.530399    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" podUID="ecfea579-6973-4647-9bd4-3edc50217db8" containerName="maas-api" containerID="cri-o://7baff75df611010160a621187918273751960239b441a78422439a95c890fb28" gracePeriod=30
Oct 31 01:06:37.825853 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.825824    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:06:37.882233 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.882203    4478 generic.go:356] "Generic (PLEG): container finished" podID="ecfea579-6973-4647-9bd4-3edc50217db8" containerID="7baff75df611010160a621187918273751960239b441a78422439a95c890fb28" exitCode=0
Oct 31 01:06:37.882429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.882251    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" event={"ID":"ecfea579-6973-4647-9bd4-3edc50217db8","Type":"ContainerDied","Data":"7baff75df611010160a621187918273751960239b441a78422439a95c890fb28"}
Oct 31 01:06:37.882429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.882260    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"
Oct 31 01:06:37.882429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.882291    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn" event={"ID":"ecfea579-6973-4647-9bd4-3edc50217db8","Type":"ContainerDied","Data":"5d92f1b8ca1d11e700803bcb12475ce45de1f907a5efb69d02b30de8109369b5"}
Oct 31 01:06:37.882429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.882314    4478 scope.go:122] "RemoveContainer" containerID="7baff75df611010160a621187918273751960239b441a78422439a95c890fb28"
Oct 31 01:06:37.892552 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.892534    4478 scope.go:122] "RemoveContainer" containerID="7baff75df611010160a621187918273751960239b441a78422439a95c890fb28"
Oct 31 01:06:37.892845 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:37.892827    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7baff75df611010160a621187918273751960239b441a78422439a95c890fb28\": container with ID starting with 7baff75df611010160a621187918273751960239b441a78422439a95c890fb28 not found: ID does not exist" containerID="7baff75df611010160a621187918273751960239b441a78422439a95c890fb28"
Oct 31 01:06:37.892908 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.892854    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7baff75df611010160a621187918273751960239b441a78422439a95c890fb28"} err="failed to get container status \"7baff75df611010160a621187918273751960239b441a78422439a95c890fb28\": rpc error: code = NotFound desc = could not find container \"7baff75df611010160a621187918273751960239b441a78422439a95c890fb28\": container with ID starting with 7baff75df611010160a621187918273751960239b441a78422439a95c890fb28 not found: ID does not exist"
Oct 31 01:06:37.983927 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.983883    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7\" (UniqueName: \"kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7\") pod \"ecfea579-6973-4647-9bd4-3edc50217db8\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") "
Oct 31 01:06:37.984122 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.984000    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\") pod \"ecfea579-6973-4647-9bd4-3edc50217db8\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") "
Oct 31 01:06:37.984122 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.984035    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca\") pod \"ecfea579-6973-4647-9bd4-3edc50217db8\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") "
Oct 31 01:06:37.984122 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.984067    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs\") pod \"ecfea579-6973-4647-9bd4-3edc50217db8\" (UID: \"ecfea579-6973-4647-9bd4-3edc50217db8\") "
Oct 31 01:06:37.985694 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.985653    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca" pod "ecfea579-6973-4647-9bd4-3edc50217db8" (UID: "ecfea579-6973-4647-9bd4-3edc50217db8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:37.987068 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.987026    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls" pod "ecfea579-6973-4647-9bd4-3edc50217db8" (UID: "ecfea579-6973-4647-9bd4-3edc50217db8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:06:37.987199 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.987178    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7" pod "ecfea579-6973-4647-9bd4-3edc50217db8" (UID: "ecfea579-6973-4647-9bd4-3edc50217db8"). InnerVolumeSpecName "kube-api-access-lrgs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:37.988587 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:37.988535    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs" pod "ecfea579-6973-4647-9bd4-3edc50217db8" (UID: "ecfea579-6973-4647-9bd4-3edc50217db8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:06:38.088064 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.087968    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrgs7\" (UniqueName: \"kubernetes.io/projected/ecfea579-6973-4647-9bd4-3edc50217db8-kube-api-access-lrgs7\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:38.088064 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.088015    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:38.088064 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.088047    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecfea579-6973-4647-9bd4-3edc50217db8-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:38.088064 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.088061    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ecfea579-6973-4647-9bd4-3edc50217db8-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:38.304063 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.304026    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"]
Oct 31 01:06:38.306610 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.306584    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-fae60c-697c69b698-wbwkn"]
Oct 31 01:06:38.918750 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:38.918703    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv" podUID="a55133d3-f8dd-436a-93f4-d6eeca18b4f0" containerName="payload-pre-processing" containerID="cri-o://e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558" gracePeriod=25
Oct 31 01:06:39.192873 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.192816    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecfea579-6973-4647-9bd4-3edc50217db8" path="/var/lib/kubelet/pods/ecfea579-6973-4647-9bd4-3edc50217db8/volumes"
Oct 31 01:06:39.255008 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.254971    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:06:39.314495 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.314464    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume\") pod \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") "
Oct 31 01:06:39.314669 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.314579    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8\" (UniqueName: \"kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8\") pod \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\" (UID: \"a55133d3-f8dd-436a-93f4-d6eeca18b4f0\") "
Oct 31 01:06:39.314849 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.314827    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume" pod "a55133d3-f8dd-436a-93f4-d6eeca18b4f0" (UID: "a55133d3-f8dd-436a-93f4-d6eeca18b4f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:39.316877 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.316854    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8" pod "a55133d3-f8dd-436a-93f4-d6eeca18b4f0" (UID: "a55133d3-f8dd-436a-93f4-d6eeca18b4f0"). InnerVolumeSpecName "kube-api-access-9s4m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:39.416724 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.416686    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:39.416724 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.416723    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4m8\" (UniqueName: \"kubernetes.io/projected/a55133d3-f8dd-436a-93f4-d6eeca18b4f0-kube-api-access-9s4m8\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:39.904703 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.904665    4478 generic.go:356] "Generic (PLEG): container finished" podID="a55133d3-f8dd-436a-93f4-d6eeca18b4f0" containerID="e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558" exitCode=0
Oct 31 01:06:39.904999 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.904742    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"
Oct 31 01:06:39.904999 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.904827    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv" event={"ID":"a55133d3-f8dd-436a-93f4-d6eeca18b4f0","Type":"ContainerDied","Data":"e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558"}
Oct 31 01:06:39.904999 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.904876    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv" event={"ID":"a55133d3-f8dd-436a-93f4-d6eeca18b4f0","Type":"ContainerDied","Data":"388ddaf45c2255912e76f9c6c26dcb5979961f51d3a405a36abb014e786ad929"}
Oct 31 01:06:39.904999 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.904905    4478 scope.go:122] "RemoveContainer" containerID="e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558"
Oct 31 01:06:39.916336 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.916311    4478 scope.go:122] "RemoveContainer" containerID="e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558"
Oct 31 01:06:39.922527 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:39.922500    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558\": container with ID starting with e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558 not found: ID does not exist" containerID="e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558"
Oct 31 01:06:39.922919 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:39.922539    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558"} err="failed to get container status \"e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558\": rpc error: code = NotFound desc = could not find container \"e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558\": container with ID starting with e912a19bac1f8b50616d824acc685dc795b94d35ffbc8a2c11c0cf64fdd92558 not found: ID does not exist"
Oct 31 01:06:40.071575 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:40.071539    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"]
Oct 31 01:06:40.120789 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:40.120757    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-fae60c-59d6d4b57f-qjkrv"]
Oct 31 01:06:41.118297 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:41.118267    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a55133d3-f8dd-436a-93f4-d6eeca18b4f0" path="/var/lib/kubelet/pods/a55133d3-f8dd-436a-93f4-d6eeca18b4f0/volumes"
Oct 31 01:06:41.970242 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:41.970197    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:06:41.970445 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:41.970213    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:06:44.759479 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:44.759443    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:06:51.969856 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:51.969820    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:06:51.970230 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:51.969870    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:06:52.600801 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:52.600766    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"]
Oct 31 01:06:53.306415 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.306393    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 01:06:53.371522 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.371500    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 01:06:53.379078 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.379058    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh\" (UniqueName: \"kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh\") pod \"205c9ed4-35da-4f91-b0dc-738abddef884\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") "
Oct 31 01:06:53.379208 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.379104    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume\") pod \"884be0a1-f3b8-4450-95a0-23de11cf4033\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") "
Oct 31 01:06:53.379208 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.379139    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt\" (UniqueName: \"kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt\") pod \"884be0a1-f3b8-4450-95a0-23de11cf4033\" (UID: \"884be0a1-f3b8-4450-95a0-23de11cf4033\") "
Oct 31 01:06:53.379208 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.379158    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume\") pod \"205c9ed4-35da-4f91-b0dc-738abddef884\" (UID: \"205c9ed4-35da-4f91-b0dc-738abddef884\") "
Oct 31 01:06:53.379562 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.379537    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume" pod "205c9ed4-35da-4f91-b0dc-738abddef884" (UID: "205c9ed4-35da-4f91-b0dc-738abddef884"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:53.379562 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.379544    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume" pod "884be0a1-f3b8-4450-95a0-23de11cf4033" (UID: "884be0a1-f3b8-4450-95a0-23de11cf4033"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:53.381355 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.381329    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh" pod "205c9ed4-35da-4f91-b0dc-738abddef884" (UID: "205c9ed4-35da-4f91-b0dc-738abddef884"). InnerVolumeSpecName "kube-api-access-w7cnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:53.381355 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.381342    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt" pod "884be0a1-f3b8-4450-95a0-23de11cf4033" (UID: "884be0a1-f3b8-4450-95a0-23de11cf4033"). InnerVolumeSpecName "kube-api-access-fjflt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:53.480173 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.480137    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7cnh\" (UniqueName: \"kubernetes.io/projected/205c9ed4-35da-4f91-b0dc-738abddef884-kube-api-access-w7cnh\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:53.480173 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.480166    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/884be0a1-f3b8-4450-95a0-23de11cf4033-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:53.480173 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.480178    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjflt\" (UniqueName: \"kubernetes.io/projected/884be0a1-f3b8-4450-95a0-23de11cf4033-kube-api-access-fjflt\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:53.480424 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:53.480187    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205c9ed4-35da-4f91-b0dc-738abddef884-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:54.080807 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.080772    4478 generic.go:356] "Generic (PLEG): container finished" podID="205c9ed4-35da-4f91-b0dc-738abddef884" containerID="ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892" exitCode=0
Oct 31 01:06:54.081120 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.081098    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" event={"ID":"205c9ed4-35da-4f91-b0dc-738abddef884","Type":"ContainerDied","Data":"ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892"}
Oct 31 01:06:54.081269 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.081255    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4" event={"ID":"205c9ed4-35da-4f91-b0dc-738abddef884","Type":"ContainerDied","Data":"8a7ce71a11db931e2fe81710e4727f9648eb6f4110fac6f47ff0a39c3ff5a08a"}
Oct 31 01:06:54.081452 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.081367    4478 scope.go:122] "RemoveContainer" containerID="ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892"
Oct 31 01:06:54.081712 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.081697    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"
Oct 31 01:06:54.121290 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.116112    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"
Oct 31 01:06:54.122027 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.121983    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" event={"ID":"884be0a1-f3b8-4450-95a0-23de11cf4033","Type":"ContainerDied","Data":"d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82"}
Oct 31 01:06:54.123120 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.123094    4478 generic.go:356] "Generic (PLEG): container finished" podID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerID="d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82" exitCode=0
Oct 31 01:06:54.123452 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.123367    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb" event={"ID":"884be0a1-f3b8-4450-95a0-23de11cf4033","Type":"ContainerDied","Data":"875da3b99280b1a339eb26e371eecda8bbf947cc3f330d3b4a4d1b754413bb90"}
Oct 31 01:06:54.137580 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.137444    4478 scope.go:122] "RemoveContainer" containerID="ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892"
Oct 31 01:06:54.137844 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:54.137823    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892\": container with ID starting with ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892 not found: ID does not exist" containerID="ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892"
Oct 31 01:06:54.137896 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.137853    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892"} err="failed to get container status \"ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892\": rpc error: code = NotFound desc = could not find container \"ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892\": container with ID starting with ad6dfc086ddfb74438477f42bbe85da0ab9ef7c3d7ef5a12d82ad6def0025892 not found: ID does not exist"
Oct 31 01:06:54.137896 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.137872    4478 scope.go:122] "RemoveContainer" containerID="d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82"
Oct 31 01:06:54.148606 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.148575    4478 scope.go:122] "RemoveContainer" containerID="d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82"
Oct 31 01:06:54.148932 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:54.148912    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82\": container with ID starting with d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82 not found: ID does not exist" containerID="d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82"
Oct 31 01:06:54.149017 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.148936    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82"} err="failed to get container status \"d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82\": rpc error: code = NotFound desc = could not find container \"d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82\": container with ID starting with d0a223d820ade3b3dc70f5f886ab5c82973eb0f9deda80b42882bd70d7036b82 not found: ID does not exist"
Oct 31 01:06:54.261491 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.261462    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"]
Oct 31 01:06:54.266839 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.266747    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-16b984-84f95b458f-sx6v4"]
Oct 31 01:06:54.386792 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.384617    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"]
Oct 31 01:06:54.393508 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:54.393468    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-16b984-6758d977c5-hxncb"]
Oct 31 01:06:55.117754 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:55.117725    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" path="/var/lib/kubelet/pods/205c9ed4-35da-4f91-b0dc-738abddef884/volumes"
Oct 31 01:06:55.118152 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:55.118139    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" path="/var/lib/kubelet/pods/884be0a1-f3b8-4450-95a0-23de11cf4033/volumes"
Oct 31 01:06:57.601946 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:57.601898    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs" podUID="b188ed46-edd2-4d5c-a08e-e744043e7f99" containerName="payload-processing" containerID="cri-o://fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a" gracePeriod=25
Oct 31 01:06:57.884720 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:57.884694    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:06:57.931208 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:57.931180    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx\" (UniqueName: \"kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx\") pod \"b188ed46-edd2-4d5c-a08e-e744043e7f99\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") "
Oct 31 01:06:57.931357 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:57.931233    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume\") pod \"b188ed46-edd2-4d5c-a08e-e744043e7f99\" (UID: \"b188ed46-edd2-4d5c-a08e-e744043e7f99\") "
Oct 31 01:06:57.931725 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:57.931697    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume" pod "b188ed46-edd2-4d5c-a08e-e744043e7f99" (UID: "b188ed46-edd2-4d5c-a08e-e744043e7f99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:06:57.933593 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:57.933561    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx" pod "b188ed46-edd2-4d5c-a08e-e744043e7f99" (UID: "b188ed46-edd2-4d5c-a08e-e744043e7f99"). InnerVolumeSpecName "kube-api-access-d29hx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:06:58.032498 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.032465    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d29hx\" (UniqueName: \"kubernetes.io/projected/b188ed46-edd2-4d5c-a08e-e744043e7f99-kube-api-access-d29hx\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:58.032498 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.032492    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b188ed46-edd2-4d5c-a08e-e744043e7f99-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:06:58.153635 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.153555    4478 generic.go:356] "Generic (PLEG): container finished" podID="b188ed46-edd2-4d5c-a08e-e744043e7f99" containerID="fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a" exitCode=0
Oct 31 01:06:58.153635 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.153600    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"
Oct 31 01:06:58.153862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.153638    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs" event={"ID":"b188ed46-edd2-4d5c-a08e-e744043e7f99","Type":"ContainerDied","Data":"fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a"}
Oct 31 01:06:58.153862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.153702    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs" event={"ID":"b188ed46-edd2-4d5c-a08e-e744043e7f99","Type":"ContainerDied","Data":"2f0efc7f07b45ebaf66068b3edae4951ff76047b3bfc0c6f3bc9cac4272a31e2"}
Oct 31 01:06:58.153862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.153732    4478 scope.go:122] "RemoveContainer" containerID="fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a"
Oct 31 01:06:58.168653 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.168637    4478 scope.go:122] "RemoveContainer" containerID="fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a"
Oct 31 01:06:58.168951 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:06:58.168924    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a\": container with ID starting with fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a not found: ID does not exist" containerID="fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a"
Oct 31 01:06:58.169023 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.168962    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a"} err="failed to get container status \"fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a\": rpc error: code = NotFound desc = could not find container \"fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a\": container with ID starting with fd63aa513f69be85a4de893af0b9814ab8967ef5a63654cefe7f506215e1bc4a not found: ID does not exist"
Oct 31 01:06:58.318570 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.318539    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"]
Oct 31 01:06:58.322686 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:58.322659    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-aaa842-54699d4bd9-mxwrs"]
Oct 31 01:06:59.117066 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:59.117029    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b188ed46-edd2-4d5c-a08e-e744043e7f99" path="/var/lib/kubelet/pods/b188ed46-edd2-4d5c-a08e-e744043e7f99/volumes"
Oct 31 01:06:59.198212 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:59.198179    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"]
Oct 31 01:06:59.310391 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:06:59.310346    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"]
Oct 31 01:07:04.199109 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:04.199049    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerName="payload-processing" containerID="cri-o://56833486f8904a204135ee30077c1a3026918b8eebc1f9cac753ebc3d21a3f16" gracePeriod=25
Oct 31 01:07:04.311125 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:04.311077    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerName="payload-pre-processing" containerID="cri-o://237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b" gracePeriod=25
Oct 31 01:07:10.906862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:10.906825    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:11.913530 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:11.913497    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:13.901174 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:13.901137    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"]
Oct 31 01:07:13.901677 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:13.901328    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" podUID="057c7160-c16b-4af3-bc50-c6719be2af5f" containerName="maas-api" containerID="cri-o://ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9" gracePeriod=30
Oct 31 01:07:14.194848 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.194824    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 01:07:14.294831 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.294268    4478 generic.go:356] "Generic (PLEG): container finished" podID="057c7160-c16b-4af3-bc50-c6719be2af5f" containerID="ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9" exitCode=0
Oct 31 01:07:14.294831 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.294344    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" event={"ID":"057c7160-c16b-4af3-bc50-c6719be2af5f","Type":"ContainerDied","Data":"ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9"}
Oct 31 01:07:14.294831 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.294416    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9" event={"ID":"057c7160-c16b-4af3-bc50-c6719be2af5f","Type":"ContainerDied","Data":"a772e6ed16fc5f8a85f6d3f3532332dfacc6321561945bb1380208716f35afbd"}
Oct 31 01:07:14.294831 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.294446    4478 scope.go:122] "RemoveContainer" containerID="ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9"
Oct 31 01:07:14.294831 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.294614    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"
Oct 31 01:07:14.312612 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.312580    4478 scope.go:122] "RemoveContainer" containerID="ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9"
Oct 31 01:07:14.313249 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:07:14.313174    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9\": container with ID starting with ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9 not found: ID does not exist" containerID="ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9"
Oct 31 01:07:14.313249 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.313212    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9"} err="failed to get container status \"ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9\": rpc error: code = NotFound desc = could not find container \"ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9\": container with ID starting with ea4c3473a875d9ea1cf89614cb29118989f005b2dff89be4e6d7a7637b2865c9 not found: ID does not exist"
Oct 31 01:07:14.331892 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.331866    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\") pod \"057c7160-c16b-4af3-bc50-c6719be2af5f\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") "
Oct 31 01:07:14.332036 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.331950    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs\") pod \"057c7160-c16b-4af3-bc50-c6719be2af5f\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") "
Oct 31 01:07:14.332036 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.332013    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc\" (UniqueName: \"kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc\") pod \"057c7160-c16b-4af3-bc50-c6719be2af5f\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") "
Oct 31 01:07:14.332181 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.332080    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca\") pod \"057c7160-c16b-4af3-bc50-c6719be2af5f\" (UID: \"057c7160-c16b-4af3-bc50-c6719be2af5f\") "
Oct 31 01:07:14.332777 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.332689    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca" pod "057c7160-c16b-4af3-bc50-c6719be2af5f" (UID: "057c7160-c16b-4af3-bc50-c6719be2af5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:14.334751 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.334417    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs" pod "057c7160-c16b-4af3-bc50-c6719be2af5f" (UID: "057c7160-c16b-4af3-bc50-c6719be2af5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:07:14.334864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.334752    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls" pod "057c7160-c16b-4af3-bc50-c6719be2af5f" (UID: "057c7160-c16b-4af3-bc50-c6719be2af5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:07:14.335180 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.335155    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc" pod "057c7160-c16b-4af3-bc50-c6719be2af5f" (UID: "057c7160-c16b-4af3-bc50-c6719be2af5f"). InnerVolumeSpecName "kube-api-access-wf6dc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:14.435201 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.433831    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:14.438615 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.438589    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/057c7160-c16b-4af3-bc50-c6719be2af5f-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:14.438754 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.438731    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf6dc\" (UniqueName: \"kubernetes.io/projected/057c7160-c16b-4af3-bc50-c6719be2af5f-kube-api-access-wf6dc\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:14.438889 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.438766    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/057c7160-c16b-4af3-bc50-c6719be2af5f-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:14.718945 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.718854    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"]
Oct 31 01:07:14.721291 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:14.721265    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-16b984-86db7d9779-m69x9"]
Oct 31 01:07:14.759592 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:07:14.759560    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:07:15.117602 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:15.117513    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="057c7160-c16b-4af3-bc50-c6719be2af5f" path="/var/lib/kubelet/pods/057c7160-c16b-4af3-bc50-c6719be2af5f/volumes"
Oct 31 01:07:20.907228 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:20.907184    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:21.913656 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:21.913620    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:07:24.365418 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.365363    4478 generic.go:356] "Generic (PLEG): container finished" podID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerID="56833486f8904a204135ee30077c1a3026918b8eebc1f9cac753ebc3d21a3f16" exitCode=0
Oct 31 01:07:24.365778 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.365425    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" event={"ID":"c1c211c5-56f7-411b-8d89-f293640b8bd5","Type":"ContainerDied","Data":"56833486f8904a204135ee30077c1a3026918b8eebc1f9cac753ebc3d21a3f16"}
Oct 31 01:07:24.467075 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.467015    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 01:07:24.551141 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.551109    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume\") pod \"c1c211c5-56f7-411b-8d89-f293640b8bd5\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") "
Oct 31 01:07:24.551331 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.551292    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v\" (UniqueName: \"kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v\") pod \"c1c211c5-56f7-411b-8d89-f293640b8bd5\" (UID: \"c1c211c5-56f7-411b-8d89-f293640b8bd5\") "
Oct 31 01:07:24.551595 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.551570    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume" pod "c1c211c5-56f7-411b-8d89-f293640b8bd5" (UID: "c1c211c5-56f7-411b-8d89-f293640b8bd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:24.551762 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.551743    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c211c5-56f7-411b-8d89-f293640b8bd5-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:24.553855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.553825    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v" pod "c1c211c5-56f7-411b-8d89-f293640b8bd5" (UID: "c1c211c5-56f7-411b-8d89-f293640b8bd5"). InnerVolumeSpecName "kube-api-access-npz9v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:24.573058 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.573038    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 01:07:24.652463 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.652430    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npz9v\" (UniqueName: \"kubernetes.io/projected/c1c211c5-56f7-411b-8d89-f293640b8bd5-kube-api-access-npz9v\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:24.753567 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.753494    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume\") pod \"a237c6d0-a8f8-4af0-84b6-82610978e14a\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") "
Oct 31 01:07:24.753732 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.753697    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8\" (UniqueName: \"kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8\") pod \"a237c6d0-a8f8-4af0-84b6-82610978e14a\" (UID: \"a237c6d0-a8f8-4af0-84b6-82610978e14a\") "
Oct 31 01:07:24.753837 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.753815    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume" pod "a237c6d0-a8f8-4af0-84b6-82610978e14a" (UID: "a237c6d0-a8f8-4af0-84b6-82610978e14a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:24.754064 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.754047    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a237c6d0-a8f8-4af0-84b6-82610978e14a-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:24.755950 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.755932    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8" pod "a237c6d0-a8f8-4af0-84b6-82610978e14a" (UID: "a237c6d0-a8f8-4af0-84b6-82610978e14a"). InnerVolumeSpecName "kube-api-access-sf6l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:24.855491 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:24.855461    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf6l8\" (UniqueName: \"kubernetes.io/projected/a237c6d0-a8f8-4af0-84b6-82610978e14a-kube-api-access-sf6l8\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:25.374137 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.374102    4478 generic.go:356] "Generic (PLEG): container finished" podID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerID="237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b" exitCode=0
Oct 31 01:07:25.374621 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.374148    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"
Oct 31 01:07:25.374621 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.374184    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" event={"ID":"a237c6d0-a8f8-4af0-84b6-82610978e14a","Type":"ContainerDied","Data":"237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b"}
Oct 31 01:07:25.374621 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.374241    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n" event={"ID":"a237c6d0-a8f8-4af0-84b6-82610978e14a","Type":"ContainerDied","Data":"93a8e3f187f1be9ff476320970f6647ebeeefd382ebc7b2c558bfd2ddca5180e"}
Oct 31 01:07:25.374621 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.374271    4478 scope.go:122] "RemoveContainer" containerID="237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b"
Oct 31 01:07:25.376216 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.375880    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w" event={"ID":"c1c211c5-56f7-411b-8d89-f293640b8bd5","Type":"ContainerDied","Data":"0f37fde1cb5dc7a6ef6da261fff7ee243880123a32e1e90f38e884b2a5005cf0"}
Oct 31 01:07:25.376216 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.375877    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"
Oct 31 01:07:25.389122 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.389100    4478 scope.go:122] "RemoveContainer" containerID="237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b"
Oct 31 01:07:25.389444 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:07:25.389413    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b\": container with ID starting with 237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b not found: ID does not exist" containerID="237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b"
Oct 31 01:07:25.389524 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.389453    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b"} err="failed to get container status \"237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b\": rpc error: code = NotFound desc = could not find container \"237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b\": container with ID starting with 237ca54a6059a5fa6243869ac11c57590a5236429650e70d683081050e85b03b not found: ID does not exist"
Oct 31 01:07:25.389524 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.389472    4478 scope.go:122] "RemoveContainer" containerID="56833486f8904a204135ee30077c1a3026918b8eebc1f9cac753ebc3d21a3f16"
Oct 31 01:07:25.538252 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.538220    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"]
Oct 31 01:07:25.589343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.589305    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-f13553-5c9c47774f-snc7w"]
Oct 31 01:07:25.697137 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.697105    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"]
Oct 31 01:07:25.701292 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:25.701259    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-f13553-c87d777c9-4vf2n"]
Oct 31 01:07:27.120372 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:27.120340    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" path="/var/lib/kubelet/pods/a237c6d0-a8f8-4af0-84b6-82610978e14a/volumes"
Oct 31 01:07:27.120963 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:27.120942    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" path="/var/lib/kubelet/pods/c1c211c5-56f7-411b-8d89-f293640b8bd5/volumes"
Oct 31 01:07:44.759015 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:07:44.758972    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:07:46.562638 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:46.562596    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"]
Oct 31 01:07:46.563561 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:46.562863    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" podUID="dc5105fc-4fac-432d-a0b7-c85a6b17e532" containerName="maas-api" containerID="cri-o://3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff" gracePeriod=30
Oct 31 01:07:46.865734 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:46.865710    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 01:07:47.014715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.014677    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca\") pod \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") "
Oct 31 01:07:47.014919 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.014740    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs\") pod \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") "
Oct 31 01:07:47.014919 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.014810    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql\" (UniqueName: \"kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql\") pod \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") "
Oct 31 01:07:47.015030 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.014995    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\") pod \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\" (UID: \"dc5105fc-4fac-432d-a0b7-c85a6b17e532\") "
Oct 31 01:07:47.015513 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.015465    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca" pod "dc5105fc-4fac-432d-a0b7-c85a6b17e532" (UID: "dc5105fc-4fac-432d-a0b7-c85a6b17e532"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:07:47.024542 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.024511    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls" pod "dc5105fc-4fac-432d-a0b7-c85a6b17e532" (UID: "dc5105fc-4fac-432d-a0b7-c85a6b17e532"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:07:47.024542 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.024533    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql" pod "dc5105fc-4fac-432d-a0b7-c85a6b17e532" (UID: "dc5105fc-4fac-432d-a0b7-c85a6b17e532"). InnerVolumeSpecName "kube-api-access-wsmql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:07:47.024704 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.024524    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs" pod "dc5105fc-4fac-432d-a0b7-c85a6b17e532" (UID: "dc5105fc-4fac-432d-a0b7-c85a6b17e532"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:07:47.120497 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.120323    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsmql\" (UniqueName: \"kubernetes.io/projected/dc5105fc-4fac-432d-a0b7-c85a6b17e532-kube-api-access-wsmql\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.120497 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.120363    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.120497 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.120404    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc5105fc-4fac-432d-a0b7-c85a6b17e532-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.120497 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.120419    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc5105fc-4fac-432d-a0b7-c85a6b17e532-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:07:47.507063 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.507032    4478 generic.go:356] "Generic (PLEG): container finished" podID="dc5105fc-4fac-432d-a0b7-c85a6b17e532" containerID="3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff" exitCode=0
Oct 31 01:07:47.507247 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.507092    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"
Oct 31 01:07:47.507247 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.507112    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" event={"ID":"dc5105fc-4fac-432d-a0b7-c85a6b17e532","Type":"ContainerDied","Data":"3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff"}
Oct 31 01:07:47.507247 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.507166    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t" event={"ID":"dc5105fc-4fac-432d-a0b7-c85a6b17e532","Type":"ContainerDied","Data":"ec1c5a6dc9540feb0396ceae59d32dde6d685d68c8a795d08e88fe807409bad5"}
Oct 31 01:07:47.507247 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.507190    4478 scope.go:122] "RemoveContainer" containerID="3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff"
Oct 31 01:07:47.517920 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.517904    4478 scope.go:122] "RemoveContainer" containerID="3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff"
Oct 31 01:07:47.518214 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:07:47.518196    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff\": container with ID starting with 3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff not found: ID does not exist" containerID="3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff"
Oct 31 01:07:47.518282 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.518221    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff"} err="failed to get container status \"3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff\": rpc error: code = NotFound desc = could not find container \"3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff\": container with ID starting with 3b36b1818b2f11674a60326470b5a4c645f5bd3297a516c0259ce7b43937f8ff not found: ID does not exist"
Oct 31 01:07:47.673187 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.673159    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"]
Oct 31 01:07:47.678891 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:47.678855    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-f13553-55b78d89d5-lf97t"]
Oct 31 01:07:49.117521 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:07:49.117494    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc5105fc-4fac-432d-a0b7-c85a6b17e532" path="/var/lib/kubelet/pods/dc5105fc-4fac-432d-a0b7-c85a6b17e532/volumes"
Oct 31 01:08:01.764972 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:01.764871    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"]
Oct 31 01:08:06.765539 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:06.765491    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2" podUID="8956368a-0204-460d-be60-2d001593ad91" containerName="payload-pre-processing" containerID="cri-o://f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08" gracePeriod=25
Oct 31 01:08:07.019034 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.018979    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:08:07.067238 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.067209    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume\") pod \"8956368a-0204-460d-be60-2d001593ad91\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") "
Oct 31 01:08:07.067466 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.067331    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx\" (UniqueName: \"kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx\") pod \"8956368a-0204-460d-be60-2d001593ad91\" (UID: \"8956368a-0204-460d-be60-2d001593ad91\") "
Oct 31 01:08:07.067680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.067653    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume" pod "8956368a-0204-460d-be60-2d001593ad91" (UID: "8956368a-0204-460d-be60-2d001593ad91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:07.069797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.069764    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx" pod "8956368a-0204-460d-be60-2d001593ad91" (UID: "8956368a-0204-460d-be60-2d001593ad91"). InnerVolumeSpecName "kube-api-access-wbcvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:07.168732 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.168695    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8956368a-0204-460d-be60-2d001593ad91-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.168732 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.168722    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbcvx\" (UniqueName: \"kubernetes.io/projected/8956368a-0204-460d-be60-2d001593ad91-kube-api-access-wbcvx\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:07.615877 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.615847    4478 generic.go:356] "Generic (PLEG): container finished" podID="8956368a-0204-460d-be60-2d001593ad91" containerID="f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08" exitCode=0
Oct 31 01:08:07.616065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.615898    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"
Oct 31 01:08:07.616065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.615908    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2" event={"ID":"8956368a-0204-460d-be60-2d001593ad91","Type":"ContainerDied","Data":"f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08"}
Oct 31 01:08:07.616065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.615950    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2" event={"ID":"8956368a-0204-460d-be60-2d001593ad91","Type":"ContainerDied","Data":"133df148bc81584b747241021ea9237eeec50edb418f3fd86fca31212d1ed013"}
Oct 31 01:08:07.616065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.615971    4478 scope.go:122] "RemoveContainer" containerID="f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08"
Oct 31 01:08:07.625492 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.625466    4478 scope.go:122] "RemoveContainer" containerID="f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08"
Oct 31 01:08:07.625811 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:07.625789    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08\": container with ID starting with f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08 not found: ID does not exist" containerID="f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08"
Oct 31 01:08:07.625864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.625820    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08"} err="failed to get container status \"f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08\": rpc error: code = NotFound desc = could not find container \"f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08\": container with ID starting with f673b7a99bd3ae80fe9baec0c56572795f041b87eb1b521aaaf6d8f63cb03e08 not found: ID does not exist"
Oct 31 01:08:07.780193 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.780157    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"]
Oct 31 01:08:07.829565 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:07.829537    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-e8808d-5c949646b7-gghb2"]
Oct 31 01:08:08.188550 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:08.188511    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"]
Oct 31 01:08:08.296417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:08.296369    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"]
Oct 31 01:08:09.117610 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:09.117578    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8956368a-0204-460d-be60-2d001593ad91" path="/var/lib/kubelet/pods/8956368a-0204-460d-be60-2d001593ad91/volumes"
Oct 31 01:08:13.189253 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:13.189215    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerName="payload-processing" containerID="cri-o://79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159" gracePeriod=25
Oct 31 01:08:13.297307 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:13.297272    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerName="payload-pre-processing" containerID="cri-o://bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848" gracePeriod=25
Oct 31 01:08:13.619819 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:13.619740    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:13.619976 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:13.619739    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:14.760748 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:14.760702    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:08:17.282146 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:17.282107    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"]
Oct 31 01:08:17.282599 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:17.282337    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" containerID="cri-o://bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328" gracePeriod=30
Oct 31 01:08:20.339970 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:20.339936    4478 patch_prober.go:29] interesting pod/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:08:20.340463 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:20.339996    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:08:22.560349 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.560295    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:08:22.663093 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663066    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663249 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663116    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663249 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663151    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663249 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663227    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663468 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663294    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663468 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663357    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663468 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663424    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663468 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663408    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:22.663468 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663458    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663727 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663509    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy\") pod \"bcfb8348-bdee-4b32-be7b-955ea83f8707\" (UID: \"bcfb8348-bdee-4b32-be7b-955ea83f8707\") "
Oct 31 01:08:22.663727 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663578    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:22.663727 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663708    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:22.663969 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663918    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:22.663969 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663957    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.664118 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663979    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.664118 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.663997    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.664118 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.664060    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:22.666090 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.666058    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:08:22.666215 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.666129    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:22.666284 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.666237    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:22.666345 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.666318    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz" pod "bcfb8348-bdee-4b32-be7b-955ea83f8707" (UID: "bcfb8348-bdee-4b32-be7b-955ea83f8707"). InnerVolumeSpecName "kube-api-access-xh9wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:22.728594 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.728566    4478 generic.go:356] "Generic (PLEG): container finished" podID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerID="bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328" exitCode=0
Oct 31 01:08:22.728743 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.728615    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"
Oct 31 01:08:22.728743 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.728623    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" event={"ID":"bcfb8348-bdee-4b32-be7b-955ea83f8707","Type":"ContainerDied","Data":"bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328"}
Oct 31 01:08:22.728743 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.728693    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7" event={"ID":"bcfb8348-bdee-4b32-be7b-955ea83f8707","Type":"ContainerDied","Data":"b9eed088d799dc2eab7281dbe793237ab7dbfcc59e16a77d573fd989e32cd6f2"}
Oct 31 01:08:22.728743 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.728714    4478 scope.go:122] "RemoveContainer" containerID="bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328"
Oct 31 01:08:22.742328 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.742278    4478 scope.go:122] "RemoveContainer" containerID="6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"
Oct 31 01:08:22.752340 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.752317    4478 scope.go:122] "RemoveContainer" containerID="bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328"
Oct 31 01:08:22.752678 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:22.752652    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328\": container with ID starting with bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328 not found: ID does not exist" containerID="bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328"
Oct 31 01:08:22.752782 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.752693    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328"} err="failed to get container status \"bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328\": rpc error: code = NotFound desc = could not find container \"bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328\": container with ID starting with bbaaca292864d1cd3c11143e68bae8c5e543a925c10f1c89abeb232017d32328 not found: ID does not exist"
Oct 31 01:08:22.752782 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.752721    4478 scope.go:122] "RemoveContainer" containerID="6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"
Oct 31 01:08:22.753055 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:22.753034    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e\": container with ID starting with 6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e not found: ID does not exist" containerID="6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"
Oct 31 01:08:22.753171 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.753060    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e"} err="failed to get container status \"6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e\": rpc error: code = NotFound desc = could not find container \"6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e\": container with ID starting with 6ff1f969b36cd3e0f45db8d68034cc927a37c4bd5c963f0f6fafd88bbb4ca96e not found: ID does not exist"
Oct 31 01:08:22.764764 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.764743    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.764855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.764765    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.764855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.764776    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.764855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.764785    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bcfb8348-bdee-4b32-be7b-955ea83f8707-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.764855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.764794    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bcfb8348-bdee-4b32-be7b-955ea83f8707-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.764855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.764803    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xh9wz\" (UniqueName: \"kubernetes.io/projected/bcfb8348-bdee-4b32-be7b-955ea83f8707-kube-api-access-xh9wz\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:22.899453 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.899358    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"]
Oct 31 01:08:22.902882 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:22.902855    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-aaa842-openshift-default-554dcb8797-n87j7"]
Oct 31 01:08:23.117270 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:23.117239    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" path="/var/lib/kubelet/pods/bcfb8348-bdee-4b32-be7b-955ea83f8707/volumes"
Oct 31 01:08:23.235975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:23.235941    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"]
Oct 31 01:08:23.236512 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:23.236284    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy" containerID="cri-o://c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032" gracePeriod=30
Oct 31 01:08:23.619033 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:23.618947    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:23.619033 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:23.618975    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:08:28.497326 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.497300    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 01:08:28.629453 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629335    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.629453 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629403    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.629453 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629439    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.629776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629515    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.629776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629564    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.629776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629630    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.629776 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629736    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.630596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629793    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.630596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629828    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:28.630596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.629853    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy\") pod \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\" (UID: \"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a\") "
Oct 31 01:08:28.630596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630140    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:28.630596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630155    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:28.630596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630369    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:28.630948 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630603    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.630948 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630627    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.630948 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630643    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.630948 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630657    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.630948 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.630867    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:28.634614 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.634501    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:28.634614 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.634554    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:28.634614 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.634538    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:08:28.634614 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.634584    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97" pod "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" (UID: "5dd3e4c3-4a5d-412e-aa59-320547bc5c6a"). InnerVolumeSpecName "kube-api-access-8wf97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:28.731793 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.731765    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.731793 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.731792    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wf97\" (UniqueName: \"kubernetes.io/projected/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-kube-api-access-8wf97\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.731975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.731802    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.731975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.731811    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.731975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.731819    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:28.765682 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.765647    4478 generic.go:356] "Generic (PLEG): container finished" podID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerID="c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032" exitCode=0
Oct 31 01:08:28.765811 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.765702    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"
Oct 31 01:08:28.765811 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.765696    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" event={"ID":"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a","Type":"ContainerDied","Data":"c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032"}
Oct 31 01:08:28.765901 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.765878    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g" event={"ID":"5dd3e4c3-4a5d-412e-aa59-320547bc5c6a","Type":"ContainerDied","Data":"ac886f7e148860f939a68e9f11e9d521afd166f303c802426f163a7b6a7de33f"}
Oct 31 01:08:28.765946 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.765918    4478 scope.go:122] "RemoveContainer" containerID="c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032"
Oct 31 01:08:28.777002 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.776985    4478 scope.go:122] "RemoveContainer" containerID="5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"
Oct 31 01:08:28.786731 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.786714    4478 scope.go:122] "RemoveContainer" containerID="c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032"
Oct 31 01:08:28.787032 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:28.787015    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032\": container with ID starting with c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032 not found: ID does not exist" containerID="c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032"
Oct 31 01:08:28.787089 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.787044    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032"} err="failed to get container status \"c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032\": rpc error: code = NotFound desc = could not find container \"c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032\": container with ID starting with c2b9a1915f1d85538a25acb42f7d9f8d236fb01964e856288791271bffd5a032 not found: ID does not exist"
Oct 31 01:08:28.787089 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.787062    4478 scope.go:122] "RemoveContainer" containerID="5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"
Oct 31 01:08:28.787291 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:28.787275    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75\": container with ID starting with 5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75 not found: ID does not exist" containerID="5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"
Oct 31 01:08:28.787346 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.787296    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75"} err="failed to get container status \"5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75\": rpc error: code = NotFound desc = could not find container \"5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75\": container with ID starting with 5ed67b5e71b3d19ce2edaed0371d2a21b088b5263e0f21fa5139606d4a2afb75 not found: ID does not exist"
Oct 31 01:08:28.930325 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.930296    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"]
Oct 31 01:08:28.935402 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:28.935365    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-4e1a6c-openshift-default-5bdff49899-6lx2g"]
Oct 31 01:08:29.118034 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:29.118003    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" path="/var/lib/kubelet/pods/5dd3e4c3-4a5d-412e-aa59-320547bc5c6a/volumes"
Oct 31 01:08:29.241063 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:29.240968    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"]
Oct 31 01:08:33.450456 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.450015    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 01:08:33.482951 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.482919    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj\" (UniqueName: \"kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj\") pod \"65a6da29-6c28-4e85-ac93-2ee7f518516e\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") "
Oct 31 01:08:33.483125 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.483097    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume\") pod \"65a6da29-6c28-4e85-ac93-2ee7f518516e\" (UID: \"65a6da29-6c28-4e85-ac93-2ee7f518516e\") "
Oct 31 01:08:33.483962 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.483924    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume" pod "65a6da29-6c28-4e85-ac93-2ee7f518516e" (UID: "65a6da29-6c28-4e85-ac93-2ee7f518516e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:33.490411 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.486480    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj" pod "65a6da29-6c28-4e85-ac93-2ee7f518516e" (UID: "65a6da29-6c28-4e85-ac93-2ee7f518516e"). InnerVolumeSpecName "kube-api-access-cblbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:33.544624 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.544595    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 01:08:33.583728 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.583701    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb\" (UniqueName: \"kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb\") pod \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") "
Oct 31 01:08:33.583880 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.583841    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume\") pod \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\" (UID: \"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc\") "
Oct 31 01:08:33.584201 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.584179    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65a6da29-6c28-4e85-ac93-2ee7f518516e-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:33.584201 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.584202    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblbj\" (UniqueName: \"kubernetes.io/projected/65a6da29-6c28-4e85-ac93-2ee7f518516e-kube-api-access-cblbj\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:33.584338 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.584262    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume" pod "8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" (UID: "8b8aacb8-6c17-4ba6-a440-cff64a9f85bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:33.585986 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.585962    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb" pod "8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" (UID: "8b8aacb8-6c17-4ba6-a440-cff64a9f85bc"). InnerVolumeSpecName "kube-api-access-np2vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:33.684928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.684898    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np2vb\" (UniqueName: \"kubernetes.io/projected/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-kube-api-access-np2vb\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:33.684928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.684927    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:33.796304 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.796219    4478 generic.go:356] "Generic (PLEG): container finished" podID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerID="79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159" exitCode=0
Oct 31 01:08:33.796304 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.796269    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"
Oct 31 01:08:33.796549 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.796307    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" event={"ID":"65a6da29-6c28-4e85-ac93-2ee7f518516e","Type":"ContainerDied","Data":"79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159"}
Oct 31 01:08:33.796549 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.796369    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl" event={"ID":"65a6da29-6c28-4e85-ac93-2ee7f518516e","Type":"ContainerDied","Data":"7b3b24d7acc9a0e5793e522d28efb88a2f986d56f2d9d29b5b19a1686a8f439c"}
Oct 31 01:08:33.796549 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.796424    4478 scope.go:122] "RemoveContainer" containerID="79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159"
Oct 31 01:08:33.797735 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.797711    4478 generic.go:356] "Generic (PLEG): container finished" podID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerID="bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848" exitCode=0
Oct 31 01:08:33.797846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.797755    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"
Oct 31 01:08:33.797846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.797793    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" event={"ID":"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc","Type":"ContainerDied","Data":"bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848"}
Oct 31 01:08:33.797974 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.797856    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl" event={"ID":"8b8aacb8-6c17-4ba6-a440-cff64a9f85bc","Type":"ContainerDied","Data":"56d997924855e81d007c932b80a99aad1ca229f1196ad0071f8bbe63db32bb56"}
Oct 31 01:08:33.806643 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.806605    4478 scope.go:122] "RemoveContainer" containerID="79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159"
Oct 31 01:08:33.806877 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:33.806858    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159\": container with ID starting with 79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159 not found: ID does not exist" containerID="79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159"
Oct 31 01:08:33.806942 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.806884    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159"} err="failed to get container status \"79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159\": rpc error: code = NotFound desc = could not find container \"79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159\": container with ID starting with 79dd91e0c9b85e6a3cf0d9f35ec71c964e20767f2db5da6cb599726ca6b67159 not found: ID does not exist"
Oct 31 01:08:33.806942 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.806900    4478 scope.go:122] "RemoveContainer" containerID="bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848"
Oct 31 01:08:33.815508 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.815492    4478 scope.go:122] "RemoveContainer" containerID="bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848"
Oct 31 01:08:33.815764 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:33.815744    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848\": container with ID starting with bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848 not found: ID does not exist" containerID="bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848"
Oct 31 01:08:33.815830 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.815770    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848"} err="failed to get container status \"bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848\": rpc error: code = NotFound desc = could not find container \"bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848\": container with ID starting with bce837937bd3caf550f188e8fcf615a8694cd1ef921e5ec0bdaeee35acf14848 not found: ID does not exist"
Oct 31 01:08:33.961797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.961759    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"]
Oct 31 01:08:33.966010 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:33.965984    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4e1a6c-8445db96c7-sdkdl"]
Oct 31 01:08:34.075342 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.075216    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"]
Oct 31 01:08:34.078483 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.078458    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4e1a6c-74b7b98f99-8lbsl"]
Oct 31 01:08:34.241804 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.241750    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn" podUID="d11ca345-48df-4668-9134-a3836787ba57" containerName="payload-pre-processing" containerID="cri-o://26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5" gracePeriod=25
Oct 31 01:08:34.544454 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.544421    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:08:34.596365 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.596338    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg\" (UniqueName: \"kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg\") pod \"d11ca345-48df-4668-9134-a3836787ba57\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") "
Oct 31 01:08:34.596523 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.596395    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume\") pod \"d11ca345-48df-4668-9134-a3836787ba57\" (UID: \"d11ca345-48df-4668-9134-a3836787ba57\") "
Oct 31 01:08:34.596855 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.596822    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume" pod "d11ca345-48df-4668-9134-a3836787ba57" (UID: "d11ca345-48df-4668-9134-a3836787ba57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:34.598646 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.598622    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg" pod "d11ca345-48df-4668-9134-a3836787ba57" (UID: "d11ca345-48df-4668-9134-a3836787ba57"). InnerVolumeSpecName "kube-api-access-qm5wg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:34.697356 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.697329    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm5wg\" (UniqueName: \"kubernetes.io/projected/d11ca345-48df-4668-9134-a3836787ba57-kube-api-access-qm5wg\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:34.697527 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.697362    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d11ca345-48df-4668-9134-a3836787ba57-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:34.804429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.804362    4478 generic.go:356] "Generic (PLEG): container finished" podID="d11ca345-48df-4668-9134-a3836787ba57" containerID="26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5" exitCode=0
Oct 31 01:08:34.804429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.804411    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn" event={"ID":"d11ca345-48df-4668-9134-a3836787ba57","Type":"ContainerDied","Data":"26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5"}
Oct 31 01:08:34.804429 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.804427    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"
Oct 31 01:08:34.804673 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.804468    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn" event={"ID":"d11ca345-48df-4668-9134-a3836787ba57","Type":"ContainerDied","Data":"438f86e1ad3ebe5d0e614bb747260d295255194e771ed4656914f8532be3699c"}
Oct 31 01:08:34.804673 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.804492    4478 scope.go:122] "RemoveContainer" containerID="26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5"
Oct 31 01:08:34.814771 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.814744    4478 scope.go:122] "RemoveContainer" containerID="26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5"
Oct 31 01:08:34.815042 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:34.815023    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5\": container with ID starting with 26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5 not found: ID does not exist" containerID="26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5"
Oct 31 01:08:34.815094 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.815054    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5"} err="failed to get container status \"26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5\": rpc error: code = NotFound desc = could not find container \"26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5\": container with ID starting with 26f9de10f04357fa7b8a7b1d05498ce963457afe4a0c6af384ea5fb9272cb3f5 not found: ID does not exist"
Oct 31 01:08:34.970586 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.970495    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"]
Oct 31 01:08:34.975000 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:34.974968    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-395b06-57797c564c-nfdcn"]
Oct 31 01:08:35.117183 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:35.117145    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" path="/var/lib/kubelet/pods/65a6da29-6c28-4e85-ac93-2ee7f518516e/volumes"
Oct 31 01:08:35.117573 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:35.117558    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" path="/var/lib/kubelet/pods/8b8aacb8-6c17-4ba6-a440-cff64a9f85bc/volumes"
Oct 31 01:08:35.117893 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:35.117882    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d11ca345-48df-4668-9134-a3836787ba57" path="/var/lib/kubelet/pods/d11ca345-48df-4668-9134-a3836787ba57/volumes"
Oct 31 01:08:40.905283 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:40.905255    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"]
Oct 31 01:08:40.905695 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:40.905496    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" podUID="7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" containerName="maas-api" containerID="cri-o://17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481" gracePeriod=30
Oct 31 01:08:41.181605 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.181576    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 01:08:41.286417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.284139    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5\" (UniqueName: \"kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5\") pod \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") "
Oct 31 01:08:41.286417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.284186    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs\") pod \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") "
Oct 31 01:08:41.286417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.284247    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca\") pod \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") "
Oct 31 01:08:41.286417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.285091    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca" pod "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" (UID: "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:41.286417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.284410    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\") pod \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\" (UID: \"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029\") "
Oct 31 01:08:41.290401 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.287555    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:41.290619 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.290576    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5" pod "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" (UID: "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029"). InnerVolumeSpecName "kube-api-access-dz6s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:41.290619 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.290611    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls" pod "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" (UID: "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:41.295428 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.295365    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs" pod "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" (UID: "7a7cc49d-8d85-4098-82aa-c1ec6c2cd029"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:08:41.388359 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.388325    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:41.388359 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.388357    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dz6s5\" (UniqueName: \"kubernetes.io/projected/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-kube-api-access-dz6s5\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:41.388565 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.388369    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:41.854565 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.854523    4478 generic.go:356] "Generic (PLEG): container finished" podID="7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" containerID="17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481" exitCode=0
Oct 31 01:08:41.854796 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.854665    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" event={"ID":"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029","Type":"ContainerDied","Data":"17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481"}
Oct 31 01:08:41.854796 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.854749    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5" event={"ID":"7a7cc49d-8d85-4098-82aa-c1ec6c2cd029","Type":"ContainerDied","Data":"ebf99eb44895f95bedb1c7df53d1d9ed5c137ee7199f56dec8502abe8b380729"}
Oct 31 01:08:41.854918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.854802    4478 scope.go:122] "RemoveContainer" containerID="17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481"
Oct 31 01:08:41.854975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.854622    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"
Oct 31 01:08:41.870488 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.870466    4478 scope.go:122] "RemoveContainer" containerID="17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481"
Oct 31 01:08:41.870958 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:41.870933    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481\": container with ID starting with 17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481 not found: ID does not exist" containerID="17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481"
Oct 31 01:08:41.871072 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:41.870971    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481"} err="failed to get container status \"17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481\": rpc error: code = NotFound desc = could not find container \"17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481\": container with ID starting with 17599a090c8c035c39358ed83401906f61aca8defa94007284b5c89c83257481 not found: ID does not exist"
Oct 31 01:08:42.020538 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:42.020506    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"]
Oct 31 01:08:42.027037 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:42.027009    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-bf325c-554548db85-572b5"]
Oct 31 01:08:43.117674 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:43.117620    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" path="/var/lib/kubelet/pods/7a7cc49d-8d85-4098-82aa-c1ec6c2cd029/volumes"
Oct 31 01:08:44.757442 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:44.757373    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:08:50.108800 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:50.108770    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"]
Oct 31 01:08:50.109176 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:50.108972    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy" containerID="cri-o://ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac" gracePeriod=30
Oct 31 01:08:55.369563 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.369533    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 01:08:55.540312 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540278    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540571 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540342    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540571 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540410    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540571 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540459    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540571 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540514    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540617    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540649    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540683    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.540797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540686    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:55.540797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.540716    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5\") pod \"94c221e4-e004-428e-bc7c-de47bf336acc\" (UID: \"94c221e4-e004-428e-bc7c-de47bf336acc\") "
Oct 31 01:08:55.541239 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.541059    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.541341 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.541239    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:55.542283 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.542124    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:08:55.542283 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.542218    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:55.542283 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.542253    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:55.544128 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.543742    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:55.544128 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.543827    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:08:55.544128 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.544027    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "kube-api-access-hcvf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:08:55.544629 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.544601    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo" pod "94c221e4-e004-428e-bc7c-de47bf336acc" (UID: "94c221e4-e004-428e-bc7c-de47bf336acc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642226    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642255    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642264    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94c221e4-e004-428e-bc7c-de47bf336acc-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642280    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642295    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcvf5\" (UniqueName: \"kubernetes.io/projected/94c221e4-e004-428e-bc7c-de47bf336acc-kube-api-access-hcvf5\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642338    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642350    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94c221e4-e004-428e-bc7c-de47bf336acc-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.642320 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.642366    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94c221e4-e004-428e-bc7c-de47bf336acc-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:08:55.932736 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.932701    4478 generic.go:356] "Generic (PLEG): container finished" podID="94c221e4-e004-428e-bc7c-de47bf336acc" containerID="ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac" exitCode=0
Oct 31 01:08:55.932928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.932750    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" event={"ID":"94c221e4-e004-428e-bc7c-de47bf336acc","Type":"ContainerDied","Data":"ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac"}
Oct 31 01:08:55.932928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.932765    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"
Oct 31 01:08:55.932928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.932811    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl" event={"ID":"94c221e4-e004-428e-bc7c-de47bf336acc","Type":"ContainerDied","Data":"c2cc0109eab8b2f9a6413f937c19f45f3f11bd4293fe1e8b72e58cd80e6ea03e"}
Oct 31 01:08:55.932928 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.932837    4478 scope.go:122] "RemoveContainer" containerID="ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac"
Oct 31 01:08:55.943093 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.943066    4478 scope.go:122] "RemoveContainer" containerID="67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"
Oct 31 01:08:55.951331 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.951306    4478 scope.go:122] "RemoveContainer" containerID="ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac"
Oct 31 01:08:55.951602 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:55.951585    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac\": container with ID starting with ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac not found: ID does not exist" containerID="ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac"
Oct 31 01:08:55.951659 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.951610    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac"} err="failed to get container status \"ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac\": rpc error: code = NotFound desc = could not find container \"ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac\": container with ID starting with ebce4536549e34ada2575a864155ffd5438ffae99e588e828e306f8bfd38ebac not found: ID does not exist"
Oct 31 01:08:55.951659 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.951628    4478 scope.go:122] "RemoveContainer" containerID="67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"
Oct 31 01:08:55.951871 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:08:55.951854    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4\": container with ID starting with 67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4 not found: ID does not exist" containerID="67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"
Oct 31 01:08:55.951911 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:55.951879    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4"} err="failed to get container status \"67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4\": rpc error: code = NotFound desc = could not find container \"67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4\": container with ID starting with 67ba8fcb0efeb071742adf97e594d3addc12850293c81454cd360132b514cff4 not found: ID does not exist"
Oct 31 01:08:56.096875 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:56.096845    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"]
Oct 31 01:08:56.101515 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:56.101489    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-bf325c-openshift-default-69d797f5dc-8flzl"]
Oct 31 01:08:57.118623 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:08:57.118595    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" path="/var/lib/kubelet/pods/94c221e4-e004-428e-bc7c-de47bf336acc/volumes"
Oct 31 01:09:02.418220 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:02.418184    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"]
Oct 31 01:09:02.527760 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:02.527467    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"]
Oct 31 01:09:02.970207 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:02.970175    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"]
Oct 31 01:09:02.970461 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:02.970438    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" podUID="a06dc4e0-6c1b-454c-b86d-84c87bcdf882" containerName="maas-api" containerID="cri-o://475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02" gracePeriod=30
Oct 31 01:09:03.260797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.260771    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:09:03.327364 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.327336    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca\") pod \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") "
Oct 31 01:09:03.327572 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.327400    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52\" (UniqueName: \"kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52\") pod \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") "
Oct 31 01:09:03.327572 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.327433    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs\") pod \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") "
Oct 31 01:09:03.327697 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.327576    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\") pod \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\" (UID: \"a06dc4e0-6c1b-454c-b86d-84c87bcdf882\") "
Oct 31 01:09:03.327777 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.327752    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca" pod "a06dc4e0-6c1b-454c-b86d-84c87bcdf882" (UID: "a06dc4e0-6c1b-454c-b86d-84c87bcdf882"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:03.327938 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.327923    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:03.329666 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.329643    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls" pod "a06dc4e0-6c1b-454c-b86d-84c87bcdf882" (UID: "a06dc4e0-6c1b-454c-b86d-84c87bcdf882"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:09:03.329740 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.329696    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52" pod "a06dc4e0-6c1b-454c-b86d-84c87bcdf882" (UID: "a06dc4e0-6c1b-454c-b86d-84c87bcdf882"). InnerVolumeSpecName "kube-api-access-x2w52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:03.329740 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.329714    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs" pod "a06dc4e0-6c1b-454c-b86d-84c87bcdf882" (UID: "a06dc4e0-6c1b-454c-b86d-84c87bcdf882"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:09:03.429023 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.428990    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:03.429023 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.429020    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2w52\" (UniqueName: \"kubernetes.io/projected/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-kube-api-access-x2w52\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:03.429023 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.429030    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a06dc4e0-6c1b-454c-b86d-84c87bcdf882-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:03.988496 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.988462    4478 generic.go:356] "Generic (PLEG): container finished" podID="a06dc4e0-6c1b-454c-b86d-84c87bcdf882" containerID="475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02" exitCode=0
Oct 31 01:09:03.988763 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.988610    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" event={"ID":"a06dc4e0-6c1b-454c-b86d-84c87bcdf882","Type":"ContainerDied","Data":"475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02"}
Oct 31 01:09:03.988763 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.988719    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2" event={"ID":"a06dc4e0-6c1b-454c-b86d-84c87bcdf882","Type":"ContainerDied","Data":"a808ae98d7ec9864da0133d00f5a4279645878134cf9bf3f1629ef893e91ce0e"}
Oct 31 01:09:03.988763 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.988752    4478 scope.go:122] "RemoveContainer" containerID="475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02"
Oct 31 01:09:03.988955 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:03.988936    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"
Oct 31 01:09:04.003214 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:04.003193    4478 scope.go:122] "RemoveContainer" containerID="475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02"
Oct 31 01:09:04.003545 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:09:04.003524    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02\": container with ID starting with 475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02 not found: ID does not exist" containerID="475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02"
Oct 31 01:09:04.003616 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:04.003555    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02"} err="failed to get container status \"475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02\": rpc error: code = NotFound desc = could not find container \"475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02\": container with ID starting with 475a2dd0b248bddf2e2cf0a798d10ff8b23cb0f8e0dc65f413faa77723279b02 not found: ID does not exist"
Oct 31 01:09:04.157237 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:04.157200    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"]
Oct 31 01:09:04.167874 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:04.165249    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-e8808d-55dc766d66-g9tl2"]
Oct 31 01:09:05.117014 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:05.116982    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a06dc4e0-6c1b-454c-b86d-84c87bcdf882" path="/var/lib/kubelet/pods/a06dc4e0-6c1b-454c-b86d-84c87bcdf882/volumes"
Oct 31 01:09:07.418540 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:07.418503    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerName="payload-processing" containerID="cri-o://a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29" gracePeriod=25
Oct 31 01:09:07.528703 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:07.528647    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerName="payload-pre-processing" containerID="cri-o://fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805" gracePeriod=25
Oct 31 01:09:14.758331 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:09:14.758292    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:09:15.865047 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:15.865010    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:09:15.865516 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:15.865050    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:09:18.549479 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:18.549444    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"]
Oct 31 01:09:25.865323 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:25.865288    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:09:25.865323 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:25.865305    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:09:27.691274 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.691204    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 01:09:27.786788 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.786757    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 01:09:27.796935 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.796907    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf\" (UniqueName: \"kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf\") pod \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") "
Oct 31 01:09:27.797047 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.797033    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume\") pod \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\" (UID: \"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563\") "
Oct 31 01:09:27.797410 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.797368    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume" pod "c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" (UID: "c8bbd7ff-2c42-4c29-94ad-ba7462e0b563"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:27.799066 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.799049    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf" pod "c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" (UID: "c8bbd7ff-2c42-4c29-94ad-ba7462e0b563"). InnerVolumeSpecName "kube-api-access-z87bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:27.897863 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.897836    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume\") pod \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") "
Oct 31 01:09:27.898043 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.897915    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m\" (UniqueName: \"kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m\") pod \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\" (UID: \"db62e589-ff36-4754-afe9-c0e7cc9dbf0e\") "
Oct 31 01:09:27.898176 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.898163    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z87bf\" (UniqueName: \"kubernetes.io/projected/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-kube-api-access-z87bf\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:27.898237 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.898177    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:27.898237 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.898178    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume" pod "db62e589-ff36-4754-afe9-c0e7cc9dbf0e" (UID: "db62e589-ff36-4754-afe9-c0e7cc9dbf0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:09:27.900212 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.900189    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m" pod "db62e589-ff36-4754-afe9-c0e7cc9dbf0e" (UID: "db62e589-ff36-4754-afe9-c0e7cc9dbf0e"). InnerVolumeSpecName "kube-api-access-gnl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:27.999747 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.999668    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnl9m\" (UniqueName: \"kubernetes.io/projected/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-kube-api-access-gnl9m\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:27.999747 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:27.999697    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/db62e589-ff36-4754-afe9-c0e7cc9dbf0e-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:28.124656 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.124626    4478 generic.go:356] "Generic (PLEG): container finished" podID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerID="fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805" exitCode=0
Oct 31 01:09:28.124810 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.124665    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"
Oct 31 01:09:28.124810 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.124699    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" event={"ID":"db62e589-ff36-4754-afe9-c0e7cc9dbf0e","Type":"ContainerDied","Data":"fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805"}
Oct 31 01:09:28.124810 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.124743    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b" event={"ID":"db62e589-ff36-4754-afe9-c0e7cc9dbf0e","Type":"ContainerDied","Data":"3a0ad78dc659f215141a309f9ec0a9a1caba2437e98157f389b8ce82bc4d4b0f"}
Oct 31 01:09:28.124810 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.124764    4478 scope.go:122] "RemoveContainer" containerID="fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805"
Oct 31 01:09:28.126105 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.126087    4478 generic.go:356] "Generic (PLEG): container finished" podID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerID="a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29" exitCode=0
Oct 31 01:09:28.126222 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.126113    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" event={"ID":"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563","Type":"ContainerDied","Data":"a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29"}
Oct 31 01:09:28.126222 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.126148    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"
Oct 31 01:09:28.126222 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.126166    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt" event={"ID":"c8bbd7ff-2c42-4c29-94ad-ba7462e0b563","Type":"ContainerDied","Data":"7c74991618f0247daf723c1918d72c9a042f0a197f1c1aab7302e34b259416c6"}
Oct 31 01:09:28.135334 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.135319    4478 scope.go:122] "RemoveContainer" containerID="fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805"
Oct 31 01:09:28.135650 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:09:28.135623    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805\": container with ID starting with fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805 not found: ID does not exist" containerID="fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805"
Oct 31 01:09:28.135728 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.135657    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805"} err="failed to get container status \"fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805\": rpc error: code = NotFound desc = could not find container \"fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805\": container with ID starting with fc4bb122449001baf5ad0380cab08c3f16f74d6de9ace2980ac09f3fdce60805 not found: ID does not exist"
Oct 31 01:09:28.135728 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.135677    4478 scope.go:122] "RemoveContainer" containerID="a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29"
Oct 31 01:09:28.143709 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.143694    4478 scope.go:122] "RemoveContainer" containerID="a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29"
Oct 31 01:09:28.144003 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:09:28.143984    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29\": container with ID starting with a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29 not found: ID does not exist" containerID="a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29"
Oct 31 01:09:28.144078 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.144008    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29"} err="failed to get container status \"a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29\": rpc error: code = NotFound desc = could not find container \"a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29\": container with ID starting with a13cf1334ec9f8cb15c91d3e065f9ca5900c6fe5c07265787e0694c84afd2d29 not found: ID does not exist"
Oct 31 01:09:28.288043 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.287976    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"]
Oct 31 01:09:28.292074 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.292051    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e1ca71-65c459b9dc-pg2pt"]
Oct 31 01:09:28.399667 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.399605    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"]
Oct 31 01:09:28.406315 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:28.406293    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e1ca71-7d979c7cf5-9l74b"]
Oct 31 01:09:29.118370 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:29.118338    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" path="/var/lib/kubelet/pods/c8bbd7ff-2c42-4c29-94ad-ba7462e0b563/volumes"
Oct 31 01:09:29.118770 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:29.118703    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" path="/var/lib/kubelet/pods/db62e589-ff36-4754-afe9-c0e7cc9dbf0e/volumes"
Oct 31 01:09:33.570189 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.570144    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main" containerID="cri-o://ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78" gracePeriod=45
Oct 31 01:09:33.855543 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.855513    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 01:09:33.972607 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.972572    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm\" (UniqueName: \"kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm\") pod \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") "
Oct 31 01:09:33.972803 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.972637    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm\") pod \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") "
Oct 31 01:09:33.972803 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.972777    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs\" (UniqueName: \"kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs\") pod \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") "
Oct 31 01:09:33.972933 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.972830    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache\") pod \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") "
Oct 31 01:09:33.972933 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.972859    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home\") pod \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") "
Oct 31 01:09:33.972933 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.972908    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir\") pod \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\" (UID: \"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2\") "
Oct 31 01:09:33.973593 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.973363    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache" pod "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" (UID: "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:33.973593 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.973486    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home" pod "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" (UID: "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:33.973802 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.973731    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir" pod "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" (UID: "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:33.976436 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.975920    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm" pod "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" (UID: "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2"). InnerVolumeSpecName "kube-api-access-4rsmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:09:33.976436 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.976003    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs" pod "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" (UID: "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:09:33.976660 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:33.976638    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm" pod "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" (UID: "6d92ea20-9c90-4a1b-a2d6-5ba98df509d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:09:34.075019 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.074978    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rsmm\" (UniqueName: \"kubernetes.io/projected/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-kube-api-access-4rsmm\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:34.075019 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.075024    4478 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-dshm\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:34.075244 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.075040    4478 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tls-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:34.075244 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.075055    4478 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-model-cache\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:34.075244 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.075071    4478 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-home\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:34.075244 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.075111    4478 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2-tmp-dir\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:09:34.161897 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.161810    4478 generic.go:356] "Generic (PLEG): container finished" podID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerID="ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78" exitCode=0
Oct 31 01:09:34.161897 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.161873    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"
Oct 31 01:09:34.162077 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.161907    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" event={"ID":"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2","Type":"ContainerDied","Data":"ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78"}
Oct 31 01:09:34.162077 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.161952    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n" event={"ID":"6d92ea20-9c90-4a1b-a2d6-5ba98df509d2","Type":"ContainerDied","Data":"7c259b25109949a61b49c23f3e4d3d0603a69a4f37e7ab94ff860f3a56ba9a97"}
Oct 31 01:09:34.162077 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.161969    4478 scope.go:122] "RemoveContainer" containerID="ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78"
Oct 31 01:09:34.171532 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.171511    4478 scope.go:122] "RemoveContainer" containerID="ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78"
Oct 31 01:09:34.171836 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:09:34.171819    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78\": container with ID starting with ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78 not found: ID does not exist" containerID="ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78"
Oct 31 01:09:34.171888 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.171844    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78"} err="failed to get container status \"ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78\": rpc error: code = NotFound desc = could not find container \"ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78\": container with ID starting with ff019e01dbdc9c026a05d4c4be149345857f8906ed6f3818407d02940c226d78 not found: ID does not exist"
Oct 31 01:09:34.340224 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.339711    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"]
Oct 31 01:09:34.341627 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:34.341606    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-869cca/distinct-2-w1-869cca-kserve-6f65c9cdbf-7jz6n"]
Oct 31 01:09:35.116530 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:35.116500    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" path="/var/lib/kubelet/pods/6d92ea20-9c90-4a1b-a2d6-5ba98df509d2/volumes"
Oct 31 01:09:41.920923 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:41.920892    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"]
Oct 31 01:09:44.758947 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:09:44.758892    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:09:46.921537 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:46.921494    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerName="payload-processing" containerID="cri-o://0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc" gracePeriod=25
Oct 31 01:09:55.805766 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:09:55.805730    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:10:05.805459 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:05.805413    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:10:06.030031 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.029999    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"]
Oct 31 01:10:06.030222 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.030200    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" podUID="dac8e284-715e-4d78-8865-31818b6749d7" containerName="maas-api" containerID="cri-o://8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54" gracePeriod=30
Oct 31 01:10:06.283012 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.282989    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 01:10:06.325010 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.324979    4478 generic.go:356] "Generic (PLEG): container finished" podID="dac8e284-715e-4d78-8865-31818b6749d7" containerID="8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54" exitCode=0
Oct 31 01:10:06.325172 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.325022    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"
Oct 31 01:10:06.325172 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.325069    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" event={"ID":"dac8e284-715e-4d78-8865-31818b6749d7","Type":"ContainerDied","Data":"8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54"}
Oct 31 01:10:06.325172 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.325124    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx" event={"ID":"dac8e284-715e-4d78-8865-31818b6749d7","Type":"ContainerDied","Data":"36838cb0b4dc2687a201b1dc645f1ec87f7a0ceb2e644dad11c6d46bc9fe42d0"}
Oct 31 01:10:06.325172 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.325148    4478 scope.go:122] "RemoveContainer" containerID="8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54"
Oct 31 01:10:06.335356 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.335336    4478 scope.go:122] "RemoveContainer" containerID="8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54"
Oct 31 01:10:06.335696 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:10:06.335671    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54\": container with ID starting with 8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54 not found: ID does not exist" containerID="8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54"
Oct 31 01:10:06.335793 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.335700    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54"} err="failed to get container status \"8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54\": rpc error: code = NotFound desc = could not find container \"8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54\": container with ID starting with 8d98ad78aed16cc1bed06649183feb6ad0be90d54439cad1e35326289309cf54 not found: ID does not exist"
Oct 31 01:10:06.399607 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.399528    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t\" (UniqueName: \"kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t\") pod \"dac8e284-715e-4d78-8865-31818b6749d7\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") "
Oct 31 01:10:06.399607 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.399570    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca\") pod \"dac8e284-715e-4d78-8865-31818b6749d7\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") "
Oct 31 01:10:06.399607 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.399595    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\") pod \"dac8e284-715e-4d78-8865-31818b6749d7\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") "
Oct 31 01:10:06.400506 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.399635    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs\") pod \"dac8e284-715e-4d78-8865-31818b6749d7\" (UID: \"dac8e284-715e-4d78-8865-31818b6749d7\") "
Oct 31 01:10:06.400506 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.400001    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca" pod "dac8e284-715e-4d78-8865-31818b6749d7" (UID: "dac8e284-715e-4d78-8865-31818b6749d7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:06.401739 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.401655    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/dac8e284-715e-4d78-8865-31818b6749d7-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:06.402962 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.402907    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs" pod "dac8e284-715e-4d78-8865-31818b6749d7" (UID: "dac8e284-715e-4d78-8865-31818b6749d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:06.403075 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.403035    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t" pod "dac8e284-715e-4d78-8865-31818b6749d7" (UID: "dac8e284-715e-4d78-8865-31818b6749d7"). InnerVolumeSpecName "kube-api-access-59k2t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:06.403507 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.403481    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls" pod "dac8e284-715e-4d78-8865-31818b6749d7" (UID: "dac8e284-715e-4d78-8865-31818b6749d7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:06.502509 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.502480    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:06.502509 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.502506    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-59k2t\" (UniqueName: \"kubernetes.io/projected/dac8e284-715e-4d78-8865-31818b6749d7-kube-api-access-59k2t\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:06.502509 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.502517    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/dac8e284-715e-4d78-8865-31818b6749d7-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:06.746237 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.746197    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"]
Oct 31 01:10:06.750092 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:06.750068    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-7abb95-7dfbcfdf7f-v9wgx"]
Oct 31 01:10:07.120753 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.120716    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dac8e284-715e-4d78-8865-31818b6749d7" path="/var/lib/kubelet/pods/dac8e284-715e-4d78-8865-31818b6749d7/volumes"
Oct 31 01:10:07.179784 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.179759    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 01:10:07.310347 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.310268    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume\") pod \"da0bec69-d5ef-496c-90d0-831f8a37651d\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") "
Oct 31 01:10:07.310347 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.310306    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2\" (UniqueName: \"kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2\") pod \"da0bec69-d5ef-496c-90d0-831f8a37651d\" (UID: \"da0bec69-d5ef-496c-90d0-831f8a37651d\") "
Oct 31 01:10:07.310864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.310834    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume" pod "da0bec69-d5ef-496c-90d0-831f8a37651d" (UID: "da0bec69-d5ef-496c-90d0-831f8a37651d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:07.313067 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.313045    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2" pod "da0bec69-d5ef-496c-90d0-831f8a37651d" (UID: "da0bec69-d5ef-496c-90d0-831f8a37651d"). InnerVolumeSpecName "kube-api-access-vj7x2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:07.332449 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.332422    4478 generic.go:356] "Generic (PLEG): container finished" podID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerID="0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc" exitCode=0
Oct 31 01:10:07.332549 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.332509    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"
Oct 31 01:10:07.332625 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.332544    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" event={"ID":"da0bec69-d5ef-496c-90d0-831f8a37651d","Type":"ContainerDied","Data":"0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc"}
Oct 31 01:10:07.332625 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.332605    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn" event={"ID":"da0bec69-d5ef-496c-90d0-831f8a37651d","Type":"ContainerDied","Data":"23ce0877f4b4b947a3a53acf9d626a1a13f6d7774b960cbe1812a111c8fdd67d"}
Oct 31 01:10:07.332737 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.332635    4478 scope.go:122] "RemoveContainer" containerID="0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc"
Oct 31 01:10:07.342071 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.342054    4478 scope.go:122] "RemoveContainer" containerID="0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc"
Oct 31 01:10:07.342321 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:10:07.342303    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc\": container with ID starting with 0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc not found: ID does not exist" containerID="0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc"
Oct 31 01:10:07.342369 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.342330    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc"} err="failed to get container status \"0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc\": rpc error: code = NotFound desc = could not find container \"0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc\": container with ID starting with 0bf4f5bd271cd16338bf4b26ec61657d48056416e6b1876848ac0fc8b6fb7ddc not found: ID does not exist"
Oct 31 01:10:07.412240 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.412210    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da0bec69-d5ef-496c-90d0-831f8a37651d-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:07.412240 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.412237    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj7x2\" (UniqueName: \"kubernetes.io/projected/da0bec69-d5ef-496c-90d0-831f8a37651d-kube-api-access-vj7x2\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:07.517065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.517030    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"]
Oct 31 01:10:07.527254 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:07.527227    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-cfc055-5875cc8dbf-qgcwn"]
Oct 31 01:10:09.116140 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:09.116096    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" path="/var/lib/kubelet/pods/da0bec69-d5ef-496c-90d0-831f8a37651d/volumes"
Oct 31 01:10:14.757964 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:10:14.757924    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:10:15.153262 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:15.153178    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"]
Oct 31 01:10:15.153479 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:15.153443    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy" containerID="cri-o://1e4a10694110d5b11058bd0d046e195194769eef095efc362d8eda02d56ab3f7" gracePeriod=30
Oct 31 01:10:17.212356 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.212310    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"]
Oct 31 01:10:17.212891 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.212580    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" podUID="3d3cbbb4-30fb-451d-b2b8-a232b26de18c" containerName="maas-api" containerID="cri-o://8380ea0a4624efc9495e4c41e7288d04b9923bbe75b14f62590810c487e5906a" gracePeriod=30
Oct 31 01:10:17.396927 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.396893    4478 generic.go:356] "Generic (PLEG): container finished" podID="3d3cbbb4-30fb-451d-b2b8-a232b26de18c" containerID="8380ea0a4624efc9495e4c41e7288d04b9923bbe75b14f62590810c487e5906a" exitCode=0
Oct 31 01:10:17.397104 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.396962    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" event={"ID":"3d3cbbb4-30fb-451d-b2b8-a232b26de18c","Type":"ContainerDied","Data":"8380ea0a4624efc9495e4c41e7288d04b9923bbe75b14f62590810c487e5906a"}
Oct 31 01:10:17.484362 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.484335    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 01:10:17.627723 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.627681    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca\") pod \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") "
Oct 31 01:10:17.627918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.627738    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45\" (UniqueName: \"kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45\") pod \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") "
Oct 31 01:10:17.627918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.627773    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs\") pod \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") "
Oct 31 01:10:17.627918 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.627809    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\") pod \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\" (UID: \"3d3cbbb4-30fb-451d-b2b8-a232b26de18c\") "
Oct 31 01:10:17.628100 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.628066    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca" pod "3d3cbbb4-30fb-451d-b2b8-a232b26de18c" (UID: "3d3cbbb4-30fb-451d-b2b8-a232b26de18c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:17.628253 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.628231    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:17.630472 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.630406    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls" pod "3d3cbbb4-30fb-451d-b2b8-a232b26de18c" (UID: "3d3cbbb4-30fb-451d-b2b8-a232b26de18c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:17.630472 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.630434    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs" pod "3d3cbbb4-30fb-451d-b2b8-a232b26de18c" (UID: "3d3cbbb4-30fb-451d-b2b8-a232b26de18c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:17.630663 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.630513    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45" pod "3d3cbbb4-30fb-451d-b2b8-a232b26de18c" (UID: "3d3cbbb4-30fb-451d-b2b8-a232b26de18c"). InnerVolumeSpecName "kube-api-access-cms45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:17.729162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.729084    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cms45\" (UniqueName: \"kubernetes.io/projected/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-kube-api-access-cms45\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:17.729162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.729119    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:17.729162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:17.729135    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d3cbbb4-30fb-451d-b2b8-a232b26de18c-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:18.403285 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:18.403248    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"
Oct 31 01:10:18.403285 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:18.403274    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb" event={"ID":"3d3cbbb4-30fb-451d-b2b8-a232b26de18c","Type":"ContainerDied","Data":"c5790159438df75bc4c9c9f21e443f7d68bc958e2b6099af675f58ba029cce21"}
Oct 31 01:10:18.403821 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:18.403339    4478 scope.go:122] "RemoveContainer" containerID="8380ea0a4624efc9495e4c41e7288d04b9923bbe75b14f62590810c487e5906a"
Oct 31 01:10:18.578235 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:18.578199    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"]
Oct 31 01:10:18.627785 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:18.627757    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-e1ca71-57f8fdd58b-jhbfb"]
Oct 31 01:10:19.116237 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:19.116201    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d3cbbb4-30fb-451d-b2b8-a232b26de18c" path="/var/lib/kubelet/pods/3d3cbbb4-30fb-451d-b2b8-a232b26de18c/volumes"
Oct 31 01:10:20.415471 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.415441    4478 generic.go:356] "Generic (PLEG): container finished" podID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerID="1e4a10694110d5b11058bd0d046e195194769eef095efc362d8eda02d56ab3f7" exitCode=0
Oct 31 01:10:20.415862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.415476    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" event={"ID":"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8","Type":"ContainerDied","Data":"1e4a10694110d5b11058bd0d046e195194769eef095efc362d8eda02d56ab3f7"}
Oct 31 01:10:20.415862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.415537    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k" event={"ID":"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8","Type":"ContainerDied","Data":"6cf0f53c2bb9415417f1c3fff2fb4b0fea593e71ff6cd4b712bfb4a3e04fa5c1"}
Oct 31 01:10:20.415862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.415552    4478 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6cf0f53c2bb9415417f1c3fff2fb4b0fea593e71ff6cd4b712bfb4a3e04fa5c1"
Oct 31 01:10:20.415862 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.415566    4478 scope.go:122] "RemoveContainer" containerID="167c390049597f8c482fed6eace24d598b98e7b93d709bdf535566f612909871"
Oct 31 01:10:20.421215 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.421181    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 01:10:20.556040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.555968    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556188 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556052    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556188 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556083    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556188 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556103    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556188 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556125    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556188 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556151    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556188 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556176    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556553 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556218    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:20.556553 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556253    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556553 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556280    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert\") pod \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\" (UID: \"dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8\") "
Oct 31 01:10:20.556553 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556473    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:20.556759 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556568    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:20.556759 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556742    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.556864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556761    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.556864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556778    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.556864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556797    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:20.556864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.556829    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:20.558556 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.558530    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:10:20.558674 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.558575    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:10:20.558674 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.558596    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:20.558674 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.558586    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp" pod "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" (UID: "dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8"). InnerVolumeSpecName "kube-api-access-mxzlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:20.657885 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.657852    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxzlp\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-kube-api-access-mxzlp\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.657885 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.657879    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.657885 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.657889    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.658114 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.657901    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.658114 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.657910    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:20.658114 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:20.657918    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:21.426440 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:21.426412    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"
Oct 31 01:10:21.707062 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:21.706977    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"]
Oct 31 01:10:21.711072 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:21.711046    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-7abb95-openshift-default-7ddb974b4c-nxx2k"]
Oct 31 01:10:23.116042 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:23.116008    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" path="/var/lib/kubelet/pods/dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8/volumes"
Oct 31 01:10:44.700630 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:44.700592    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"]
Oct 31 01:10:44.756922 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:10:44.756885    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:10:48.296348 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.296312    4478 scope.go:122] "RemoveContainer" containerID="1e4a10694110d5b11058bd0d046e195194769eef095efc362d8eda02d56ab3f7"
Oct 31 01:10:48.954065 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954031    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"]
Oct 31 01:10:48.954469 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954456    4478 cpu_manager.go:414] "containerMap: removing container" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerName="payload-processing"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954470    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerName="payload-processing"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954480    4478 cpu_manager.go:414] "containerMap: removing container" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954485    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954495    4478 cpu_manager.go:414] "containerMap: removing container" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954500    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954507    4478 cpu_manager.go:414] "containerMap: removing container" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerName="payload-pre-processing"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954511    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerName="payload-pre-processing"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954520    4478 cpu_manager.go:414] "containerMap: removing container" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerName="payload-pre-processing"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954525    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerName="payload-pre-processing"
Oct 31 01:10:48.954528 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954531    4478 cpu_manager.go:414] "containerMap: removing container" podUID="dc5105fc-4fac-432d-a0b7-c85a6b17e532" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954537    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc5105fc-4fac-432d-a0b7-c85a6b17e532" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954543    4478 cpu_manager.go:414] "containerMap: removing container" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954547    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954553    4478 cpu_manager.go:414] "containerMap: removing container" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954557    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954563    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b188ed46-edd2-4d5c-a08e-e744043e7f99" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954568    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b188ed46-edd2-4d5c-a08e-e744043e7f99" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954573    4478 cpu_manager.go:414] "containerMap: removing container" podUID="0956f55c-2fc4-4f39-baed-0232d37f4265" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954578    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0956f55c-2fc4-4f39-baed-0232d37f4265" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954585    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d11ca345-48df-4668-9134-a3836787ba57" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954589    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d11ca345-48df-4668-9134-a3836787ba57" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954598    4478 cpu_manager.go:414] "containerMap: removing container" podUID="033bc0df-a025-4649-8042-0644a89a9102" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954603    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="033bc0df-a025-4649-8042-0644a89a9102" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954611    4478 cpu_manager.go:414] "containerMap: removing container" podUID="dac8e284-715e-4d78-8865-31818b6749d7" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954614    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dac8e284-715e-4d78-8865-31818b6749d7" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954623    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954626    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954636    4478 cpu_manager.go:414] "containerMap: removing container" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954643    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954649    4478 cpu_manager.go:414] "containerMap: removing container" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954654    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954661    4478 cpu_manager.go:414] "containerMap: removing container" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954665    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954672    4478 cpu_manager.go:414] "containerMap: removing container" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954676    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954681    4478 cpu_manager.go:414] "containerMap: removing container" podUID="22810094-24e8-4b47-9b86-b70056819c27" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954686    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22810094-24e8-4b47-9b86-b70056819c27" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954691    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a55133d3-f8dd-436a-93f4-d6eeca18b4f0" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954696    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a55133d3-f8dd-436a-93f4-d6eeca18b4f0" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954709    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954714    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954721    4478 cpu_manager.go:414] "containerMap: removing container" podUID="3d3cbbb4-30fb-451d-b2b8-a232b26de18c" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954726    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d3cbbb4-30fb-451d-b2b8-a232b26de18c" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954731    4478 cpu_manager.go:414] "containerMap: removing container" podUID="8956368a-0204-460d-be60-2d001593ad91" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954736    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8956368a-0204-460d-be60-2d001593ad91" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954741    4478 cpu_manager.go:414] "containerMap: removing container" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954745    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954750    4478 cpu_manager.go:414] "containerMap: removing container" podUID="057c7160-c16b-4af3-bc50-c6719be2af5f" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954754    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="057c7160-c16b-4af3-bc50-c6719be2af5f" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954761    4478 cpu_manager.go:414] "containerMap: removing container" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954765    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerName="payload-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954771    4478 cpu_manager.go:414] "containerMap: removing container" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954775    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954788    4478 cpu_manager.go:414] "containerMap: removing container" podUID="7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954793    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954799    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954803    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerName="payload-pre-processing"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954809    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a06dc4e0-6c1b-454c-b86d-84c87bcdf882" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954813    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a06dc4e0-6c1b-454c-b86d-84c87bcdf882" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954818    4478 cpu_manager.go:414] "containerMap: removing container" podUID="ecfea579-6973-4647-9bd4-3edc50217db8" containerName="maas-api"
Oct 31 01:10:48.954846 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954822    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecfea579-6973-4647-9bd4-3edc50217db8" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954876    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="8956368a-0204-460d-be60-2d001593ad91" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954882    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="205c9ed4-35da-4f91-b0dc-738abddef884" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954888    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="057c7160-c16b-4af3-bc50-c6719be2af5f" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954894    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="65a6da29-6c28-4e85-ac93-2ee7f518516e" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954898    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954903    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7cc49d-8d85-4098-82aa-c1ec6c2cd029" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954908    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a237c6d0-a8f8-4af0-84b6-82610978e14a" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954912    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954917    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a06dc4e0-6c1b-454c-b86d-84c87bcdf882" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954922    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecfea579-6973-4647-9bd4-3edc50217db8" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954927    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0bec69-d5ef-496c-90d0-831f8a37651d" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954932    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d92ea20-9c90-4a1b-a2d6-5ba98df509d2" containerName="main"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954937    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954941    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b8aacb8-6c17-4ba6-a440-cff64a9f85bc" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954946    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="db62e589-ff36-4754-afe9-c0e7cc9dbf0e" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954950    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc5105fc-4fac-432d-a0b7-c85a6b17e532" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954954    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954959    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2e42da-51b1-40a8-bbf9-56d1bdbd5dc8" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954963    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b188ed46-edd2-4d5c-a08e-e744043e7f99" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954969    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="0956f55c-2fc4-4f39-baed-0232d37f4265" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954973    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d11ca345-48df-4668-9134-a3836787ba57" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954979    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="033bc0df-a025-4649-8042-0644a89a9102" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954984    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="dac8e284-715e-4d78-8865-31818b6749d7" containerName="maas-api"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954988    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c211c5-56f7-411b-8d89-f293640b8bd5" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954993    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="884be0a1-f3b8-4450-95a0-23de11cf4033" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.954998    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dd3e4c3-4a5d-412e-aa59-320547bc5c6a" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.955002    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcfb8348-bdee-4b32-be7b-955ea83f8707" containerName="istio-proxy"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.955008    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="22810094-24e8-4b47-9b86-b70056819c27" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.955012    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a55133d3-f8dd-436a-93f4-d6eeca18b4f0" containerName="payload-pre-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.955019    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8bbd7ff-2c42-4c29-94ad-ba7462e0b563" containerName="payload-processing"
Oct 31 01:10:48.956343 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.955024    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d3cbbb4-30fb-451d-b2b8-a232b26de18c" containerName="maas-api"
Oct 31 01:10:48.959729 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:48.959709    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.018318 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.018289    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"agc-cred-2e3729eb8072\""
Oct 31 01:10:49.020969 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.020932    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"]
Oct 31 01:10:49.035872 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.035842    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.035983 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.035880    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.036031 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.036006    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.036152 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.036134    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24gwk\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.088615 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.088587    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"openshift-service-ca.crt\""
Oct 31 01:10:49.088741 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.088634    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d7412d\""
Oct 31 01:10:49.088741 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.088691    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"routing-overlay\""
Oct 31 01:10:49.088741 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.088718    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"kube-root-ca.crt\""
Oct 31 01:10:49.137546 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.137515    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24gwk\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.137654 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.137597    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.137654 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.137627    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.137750 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.137676    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.138422 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.138373    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.138533 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.138441    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.140272 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.140249    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.197802 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.197780    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24gwk\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.272492 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.272426    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:49.647020 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.646674    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"]
Oct 31 01:10:49.649712 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:10:49.649680    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeab15438_e78d_42b5_ad77_8ec01199555d.slice/crio-1c2aa34c50338f5dbf1fa9fd3651e28d36db9268c8446969b72520de004374f4 WatchSource:0}: Error finding container 1c2aa34c50338f5dbf1fa9fd3651e28d36db9268c8446969b72520de004374f4: Status 404 returned error can't find the container with id 1c2aa34c50338f5dbf1fa9fd3651e28d36db9268c8446969b72520de004374f4
Oct 31 01:10:49.701763 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.701716    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerName="payload-pre-processing" containerID="cri-o://86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7" gracePeriod=25
Oct 31 01:10:49.928716 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.928681    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"]
Oct 31 01:10:49.928903 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:49.928882    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" podUID="f5fa99f3-efc9-4179-9413-de74dbf122df" containerName="maas-api" containerID="cri-o://77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080" gracePeriod=30
Oct 31 01:10:50.202008 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.201985    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 01:10:50.258711 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.258687    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p\" (UniqueName: \"kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p\") pod \"f5fa99f3-efc9-4179-9413-de74dbf122df\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") "
Oct 31 01:10:50.258868 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.258724    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\") pod \"f5fa99f3-efc9-4179-9413-de74dbf122df\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") "
Oct 31 01:10:50.258868 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.258757    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs\") pod \"f5fa99f3-efc9-4179-9413-de74dbf122df\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") "
Oct 31 01:10:50.258868 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.258788    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca\") pod \"f5fa99f3-efc9-4179-9413-de74dbf122df\" (UID: \"f5fa99f3-efc9-4179-9413-de74dbf122df\") "
Oct 31 01:10:50.259250 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.259217    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca" pod "f5fa99f3-efc9-4179-9413-de74dbf122df" (UID: "f5fa99f3-efc9-4179-9413-de74dbf122df"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:10:50.261094 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.261067    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs" pod "f5fa99f3-efc9-4179-9413-de74dbf122df" (UID: "f5fa99f3-efc9-4179-9413-de74dbf122df"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:50.261472 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.261445    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p" pod "f5fa99f3-efc9-4179-9413-de74dbf122df" (UID: "f5fa99f3-efc9-4179-9413-de74dbf122df"). InnerVolumeSpecName "kube-api-access-qsj4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:10:50.261590 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.261513    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls" pod "f5fa99f3-efc9-4179-9413-de74dbf122df" (UID: "f5fa99f3-efc9-4179-9413-de74dbf122df"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:10:50.360030 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.359986    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.360030 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.360022    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fa99f3-efc9-4179-9413-de74dbf122df-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.360030 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.360037    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qsj4p\" (UniqueName: \"kubernetes.io/projected/f5fa99f3-efc9-4179-9413-de74dbf122df-kube-api-access-qsj4p\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.360341 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.360051    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5fa99f3-efc9-4179-9413-de74dbf122df-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:10:50.600165 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.600081    4478 generic.go:356] "Generic (PLEG): container finished" podID="f5fa99f3-efc9-4179-9413-de74dbf122df" containerID="77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080" exitCode=0
Oct 31 01:10:50.600165 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.600134    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"
Oct 31 01:10:50.600436 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.600130    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" event={"ID":"f5fa99f3-efc9-4179-9413-de74dbf122df","Type":"ContainerDied","Data":"77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080"}
Oct 31 01:10:50.600436 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.600274    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz" event={"ID":"f5fa99f3-efc9-4179-9413-de74dbf122df","Type":"ContainerDied","Data":"0876e88fbe17bbf8e45b858918fcd2e3c739296e4157831e1e481c504ad43ef2"}
Oct 31 01:10:50.600436 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.600316    4478 scope.go:122] "RemoveContainer" containerID="77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080"
Oct 31 01:10:50.601904 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.601882    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9" event={"ID":"eab15438-e78d-42b5-ad77-8ec01199555d","Type":"ContainerStarted","Data":"df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5"}
Oct 31 01:10:50.602010 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.601927    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9" event={"ID":"eab15438-e78d-42b5-ad77-8ec01199555d","Type":"ContainerStarted","Data":"1c2aa34c50338f5dbf1fa9fd3651e28d36db9268c8446969b72520de004374f4"}
Oct 31 01:10:50.610482 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.610462    4478 scope.go:122] "RemoveContainer" containerID="77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080"
Oct 31 01:10:50.610746 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:10:50.610728    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080\": container with ID starting with 77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080 not found: ID does not exist" containerID="77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080"
Oct 31 01:10:50.610819 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.610753    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080"} err="failed to get container status \"77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080\": rpc error: code = NotFound desc = could not find container \"77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080\": container with ID starting with 77e3968d73a5c73219a2909aea491fddecf0172261d6ffbff5e27cb04c49e080 not found: ID does not exist"
Oct 31 01:10:50.764255 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.764210    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"]
Oct 31 01:10:50.767120 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.767095    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-869cca-6974644485-7gjnz"]
Oct 31 01:10:50.883499 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:50.883375    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9" podStartSLOduration=2.883360716 podStartE2EDuration="2.883360716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:10:50.882082535 +0000 UTC m=+4083.898747259" watchObservedRunningTime="2026-10-31 01:10:50.883360716 +0000 UTC m=+4083.900025428"
Oct 31 01:10:51.116507 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:51.116475    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fa99f3-efc9-4179-9413-de74dbf122df" path="/var/lib/kubelet/pods/f5fa99f3-efc9-4179-9413-de74dbf122df/volumes"
Oct 31 01:10:51.607348 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:51.607317    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:10:58.489959 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:58.489882    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:10:59.233641 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:59.233603    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"]
Oct 31 01:10:59.233840 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:10:59.233807    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" containerID="cri-o://dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff" gracePeriod=30
Oct 31 01:11:01.068102 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:01.068068    4478 patch_prober.go:29] interesting pod/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 31 01:11:01.068492 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:01.068126    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 31 01:11:02.615817 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:02.615788    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:11:04.497149 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.497124    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 01:11:04.599966 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.599883    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.599966 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.599922    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.599966 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.599953    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600129    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600187    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:04.600256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600226    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:04.600256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600239    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600264    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600321    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600353    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600365    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:04.600486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600417    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy\") pod \"d4037f26-6727-4523-b6b5-6c7af9a5f771\" (UID: \"d4037f26-6727-4523-b6b5-6c7af9a5f771\") "
Oct 31 01:11:04.600755 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600556    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:04.600755 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.600698    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:04.601053 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.601027    4478 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-data\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.601192 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.601056    4478 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4037f26-6727-4523-b6b5-6c7af9a5f771-istiod-ca-cert\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.601192 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.601071    4478 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-credential-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.601192 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.601085    4478 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.601192 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.601100    4478 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-workload-socket\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.602458 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.602432    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 31 01:11:04.602550 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.602518    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:04.602618 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.602595    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:04.602706 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.602685    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq" pod "d4037f26-6727-4523-b6b5-6c7af9a5f771" (UID: "d4037f26-6727-4523-b6b5-6c7af9a5f771"). InnerVolumeSpecName "kube-api-access-lspjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:04.686514 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.686472    4478 generic.go:356] "Generic (PLEG): container finished" podID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerID="dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff" exitCode=0
Oct 31 01:11:04.686699 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.686583    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"
Oct 31 01:11:04.686699 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.686614    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" event={"ID":"d4037f26-6727-4523-b6b5-6c7af9a5f771","Type":"ContainerDied","Data":"dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff"}
Oct 31 01:11:04.686834 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.686706    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t" event={"ID":"d4037f26-6727-4523-b6b5-6c7af9a5f771","Type":"ContainerDied","Data":"ed1f7591de7fe5cc69a55eb1457510370d58f3142cf1edece94df2df1f7b9aa4"}
Oct 31 01:11:04.686834 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.686737    4478 scope.go:122] "RemoveContainer" containerID="dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff"
Oct 31 01:11:04.695971 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.695955    4478 scope.go:122] "RemoveContainer" containerID="dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff"
Oct 31 01:11:04.696285 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:04.696268    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff\": container with ID starting with dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff not found: ID does not exist" containerID="dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff"
Oct 31 01:11:04.696331 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.696293    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff"} err="failed to get container status \"dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff\": rpc error: code = NotFound desc = could not find container \"dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff\": container with ID starting with dae82613e68ae24c620f0eb6ab2069751385e770fa3336380c8e3faaf900beff not found: ID does not exist"
Oct 31 01:11:04.701753 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.701732    4478 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-podinfo\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.701753 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.701753    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lspjq\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-kube-api-access-lspjq\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.701880 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.701763    4478 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-envoy\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.701880 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.701771    4478 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4037f26-6727-4523-b6b5-6c7af9a5f771-istio-token\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:04.852679 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.852585    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"]
Oct 31 01:11:04.856596 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:04.856568    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-869cca-openshift-default-86459965cf-mkh8t"]
Oct 31 01:11:05.116229 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:05.116154    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" path="/var/lib/kubelet/pods/d4037f26-6727-4523-b6b5-6c7af9a5f771/volumes"
Oct 31 01:11:08.490425 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:08.490375    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:11:09.965278 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:09.965220    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 01:11:10.047805 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.047771    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn\") pod \"990aab48-fa5c-48d2-be02-6c48dd0431a6\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") "
Oct 31 01:11:10.047805 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.047810    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume\") pod \"990aab48-fa5c-48d2-be02-6c48dd0431a6\" (UID: \"990aab48-fa5c-48d2-be02-6c48dd0431a6\") "
Oct 31 01:11:10.048254 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.048233    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume" pod "990aab48-fa5c-48d2-be02-6c48dd0431a6" (UID: "990aab48-fa5c-48d2-be02-6c48dd0431a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:10.050011 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.049988    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn" pod "990aab48-fa5c-48d2-be02-6c48dd0431a6" (UID: "990aab48-fa5c-48d2-be02-6c48dd0431a6"). InnerVolumeSpecName "kube-api-access-pvlzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:10.148819 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.148786    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvlzn\" (UniqueName: \"kubernetes.io/projected/990aab48-fa5c-48d2-be02-6c48dd0431a6-kube-api-access-pvlzn\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:10.148819 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.148820    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/990aab48-fa5c-48d2-be02-6c48dd0431a6-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:10.716617 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.716580    4478 generic.go:356] "Generic (PLEG): container finished" podID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerID="86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7" exitCode=0
Oct 31 01:11:10.716792 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.716644    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"
Oct 31 01:11:10.716792 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.716668    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" event={"ID":"990aab48-fa5c-48d2-be02-6c48dd0431a6","Type":"ContainerDied","Data":"86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7"}
Oct 31 01:11:10.716792 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.716754    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c" event={"ID":"990aab48-fa5c-48d2-be02-6c48dd0431a6","Type":"ContainerDied","Data":"ae190ab53991514d143a798593686fec02b24217770c5f9b09d157f0b5944689"}
Oct 31 01:11:10.716792 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.716782    4478 scope.go:122] "RemoveContainer" containerID="86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7"
Oct 31 01:11:10.726154 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.726139    4478 scope.go:122] "RemoveContainer" containerID="86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7"
Oct 31 01:11:10.726430 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:10.726411    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7\": container with ID starting with 86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7 not found: ID does not exist" containerID="86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7"
Oct 31 01:11:10.726508 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.726437    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7"} err="failed to get container status \"86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7\": rpc error: code = NotFound desc = could not find container \"86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7\": container with ID starting with 86b5728b63d0a4afde320562ba4993b49ec2e0d59effeedf2c6dd002f87360d7 not found: ID does not exist"
Oct 31 01:11:10.879250 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.879221    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"]
Oct 31 01:11:10.928517 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:10.928481    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c6b3b9-8bd66b4c-t248c"]
Oct 31 01:11:11.116494 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:11.116422    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" path="/var/lib/kubelet/pods/990aab48-fa5c-48d2-be02-6c48dd0431a6/volumes"
Oct 31 01:11:14.756907 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:14.756871    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:11:35.165462 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.165428    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"]
Oct 31 01:11:35.195617 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.195588    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"]
Oct 31 01:11:35.195993 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.195980    4478 cpu_manager.go:414] "containerMap: removing container" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerName="payload-pre-processing"
Oct 31 01:11:35.196040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.195994    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerName="payload-pre-processing"
Oct 31 01:11:35.196040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196005    4478 cpu_manager.go:414] "containerMap: removing container" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy"
Oct 31 01:11:35.196040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196010    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy"
Oct 31 01:11:35.196040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196019    4478 cpu_manager.go:414] "containerMap: removing container" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy"
Oct 31 01:11:35.196040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196024    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy"
Oct 31 01:11:35.196040 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196039    4478 cpu_manager.go:414] "containerMap: removing container" podUID="f5fa99f3-efc9-4179-9413-de74dbf122df" containerName="maas-api"
Oct 31 01:11:35.196232 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196044    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fa99f3-efc9-4179-9413-de74dbf122df" containerName="maas-api"
Oct 31 01:11:35.196232 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196100    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="990aab48-fa5c-48d2-be02-6c48dd0431a6" containerName="payload-pre-processing"
Oct 31 01:11:35.196232 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196107    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4037f26-6727-4523-b6b5-6c7af9a5f771" containerName="istio-proxy"
Oct 31 01:11:35.196232 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196116    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fa99f3-efc9-4179-9413-de74dbf122df" containerName="maas-api"
Oct 31 01:11:35.196232 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.196122    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c221e4-e004-428e-bc7c-de47bf336acc" containerName="istio-proxy"
Oct 31 01:11:35.199058 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.199043    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.251136 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.251108    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"agc-cred-89de3a35c154\""
Oct 31 01:11:35.251269 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.251260    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d7412d\"/\"payload-processing-external-model-e2e--80db92a2-dockercfg-gcg8t\""
Oct 31 01:11:35.277826 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.277794    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"]
Oct 31 01:11:35.279447 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.279421    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.279573 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.279457    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.279573 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.279485    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wswk\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.279690 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.279635    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.381124 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.381094    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.381327 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.381208    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.381327 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.381243    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.381327 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.381280    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wswk\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.381976 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.381944    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.382241 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.382033    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.384159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.384131    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.442467 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.442440    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wswk\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.511994 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.511970    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:35.891008 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:35.890984    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"]
Oct 31 01:11:35.893813 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:11:35.893781    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e1e103_4888_46a9_87c1_b73ae7cad0d1.slice/crio-9457f5a12e23367df5c50f49780253264d20fa846698bc6dc97baaf47748c674 WatchSource:0}: Error finding container 9457f5a12e23367df5c50f49780253264d20fa846698bc6dc97baaf47748c674: Status 404 returned error can't find the container with id 9457f5a12e23367df5c50f49780253264d20fa846698bc6dc97baaf47748c674
Oct 31 01:11:36.845713 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:36.845673    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" event={"ID":"13e1e103-4888-46a9-87c1-b73ae7cad0d1","Type":"ContainerStarted","Data":"1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3"}
Oct 31 01:11:36.846212 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:36.845882    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:36.846212 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:36.845930    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" event={"ID":"13e1e103-4888-46a9-87c1-b73ae7cad0d1","Type":"ContainerStarted","Data":"9457f5a12e23367df5c50f49780253264d20fa846698bc6dc97baaf47748c674"}
Oct 31 01:11:36.962142 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:36.962096    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" podStartSLOduration=1.962081885 podStartE2EDuration="1.962081885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:36.961057655 +0000 UTC m=+4129.977722379" watchObservedRunningTime="2026-10-31 01:11:36.962081885 +0000 UTC m=+4129.978746599"
Oct 31 01:11:39.037701 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.037667    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"]
Oct 31 01:11:39.041221 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.041198    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.092960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.092928    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3853b6\"/\"routing-overlay\""
Oct 31 01:11:39.092960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.092928    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3853b6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-3853b6\""
Oct 31 01:11:39.093209 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.093134    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3853b6\"/\"kube-root-ca.crt\""
Oct 31 01:11:39.093209 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.093156    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-3853b6\"/\"openshift-service-ca.crt\""
Oct 31 01:11:39.120155 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.120107    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.120334 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.120170    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.120334 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.120287    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hb6fr\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.120508 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.120347    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.142643 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.142614    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3853b6\"/\"agc-cred-151278ee7ee5\""
Oct 31 01:11:39.142804 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.142693    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-3853b6\"/\"payload-processing-external-model-e2e--ab9e077e-dockercfg-sgxfg\""
Oct 31 01:11:39.144551 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.144517    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"]
Oct 31 01:11:39.221302 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.221274    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.221302 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.221312    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.221525 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.221346    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hb6fr\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.221525 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.221415    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.222077 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.222056    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.222131 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.222090    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.223874 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.223853    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.282281 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.282250    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hb6fr\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr\") pod \"payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") " pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.354372 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.354293    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.730021 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.729998    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"]
Oct 31 01:11:39.732825 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:11:39.732786    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1759600_1d1e_46a2_b37b_ed2187c5df0f.slice/crio-64ccdb78b4cd10e5cbf82a759ab1ec73882ccecfbb2d978cd5d7ef12e8d6a3b3 WatchSource:0}: Error finding container 64ccdb78b4cd10e5cbf82a759ab1ec73882ccecfbb2d978cd5d7ef12e8d6a3b3: Status 404 returned error can't find the container with id 64ccdb78b4cd10e5cbf82a759ab1ec73882ccecfbb2d978cd5d7ef12e8d6a3b3
Oct 31 01:11:39.863207 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.863163    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" event={"ID":"a1759600-1d1e-46a2-b37b-ed2187c5df0f","Type":"ContainerStarted","Data":"131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558"}
Oct 31 01:11:39.863421 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.863238    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" event={"ID":"a1759600-1d1e-46a2-b37b-ed2187c5df0f","Type":"ContainerStarted","Data":"64ccdb78b4cd10e5cbf82a759ab1ec73882ccecfbb2d978cd5d7ef12e8d6a3b3"}
Oct 31 01:11:39.863421 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.863270    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:39.980418 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:39.980285    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" podStartSLOduration=1.980271474 podStartE2EDuration="1.980271474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:11:39.978870738 +0000 UTC m=+4132.995535477" watchObservedRunningTime="2026-10-31 01:11:39.980271474 +0000 UTC m=+4132.996936186"
Oct 31 01:11:40.948628 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:40.948596    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3853b6/agc-cred-151278ee7ee5: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:40.948628 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:40.948621    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:40.949110 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:40.948682    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials podName:a1759600-1d1e-46a2-b37b-ed2187c5df0f nodeName:}" failed. No retries permitted until 2026-10-31 01:11:41.448661202 +0000 UTC m=+4134.465325913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f") : secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:41.453108 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:41.453075    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3853b6/agc-cred-151278ee7ee5: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:41.453108 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:41.453103    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:41.453402 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:41.453154    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials podName:a1759600-1d1e-46a2-b37b-ed2187c5df0f nodeName:}" failed. No retries permitted until 2026-10-31 01:11:42.453140049 +0000 UTC m=+4135.469804740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f") : secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:42.464146 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:42.464116    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3853b6/agc-cred-151278ee7ee5: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:42.464146 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:42.464141    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:42.464618 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:42.464196    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials podName:a1759600-1d1e-46a2-b37b-ed2187c5df0f nodeName:}" failed. No retries permitted until 2026-10-31 01:11:44.464178501 +0000 UTC m=+4137.480843192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f") : secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:44.179176 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:44.179141    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"]
Oct 31 01:11:44.181728 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:44.181703    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:44.484956 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:44.484870    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3853b6/agc-cred-151278ee7ee5: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:44.484956 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:44.484896    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:44.484956 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:44.484949    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials podName:a1759600-1d1e-46a2-b37b-ed2187c5df0f nodeName:}" failed. No retries permitted until 2026-10-31 01:11:48.484933929 +0000 UTC m=+4141.501598638 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f") : secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:44.772837 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:44.772754    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:11:47.853252 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:47.853226    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:11:48.021069 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:48.021037    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"]
Oct 31 01:11:48.521116 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:48.521088    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-3853b6/agc-cred-151278ee7ee5: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:48.521116 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:48.521111    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc: secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:48.521310 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:48.521161    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials podName:a1759600-1d1e-46a2-b37b-ed2187c5df0f nodeName:}" failed. No retries permitted until 2026-10-31 01:11:56.521147851 +0000 UTC m=+4149.537812541 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f") : secret "agc-cred-151278ee7ee5" not found
Oct 31 01:11:49.180008 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.179971    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" podUID="a1759600-1d1e-46a2-b37b-ed2187c5df0f" containerName="payload-processing" containerID="cri-o://131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558" gracePeriod=25
Oct 31 01:11:49.431011 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.430953    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:49.528483 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.528452    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials\") pod \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") "
Oct 31 01:11:49.528634 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.528521    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume\") pod \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") "
Oct 31 01:11:49.528634 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.528552    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr\") pod \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") "
Oct 31 01:11:49.528754 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.528680    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume\") pod \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\" (UID: \"a1759600-1d1e-46a2-b37b-ed2187c5df0f\") "
Oct 31 01:11:49.528954 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.528879    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume" pod "a1759600-1d1e-46a2-b37b-ed2187c5df0f" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:49.529092 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.529019    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume" pod "a1759600-1d1e-46a2-b37b-ed2187c5df0f" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:49.530836 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.530815    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials" pod "a1759600-1d1e-46a2-b37b-ed2187c5df0f" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:49.530939 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.530838    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr" pod "a1759600-1d1e-46a2-b37b-ed2187c5df0f" (UID: "a1759600-1d1e-46a2-b37b-ed2187c5df0f"). InnerVolumeSpecName "kube-api-access-hb6fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:49.629875 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.629845    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:49.629875 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.629872    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hb6fr\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-kube-api-access-hb6fr\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:49.630061 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.629883    4478 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a1759600-1d1e-46a2-b37b-ed2187c5df0f-routing-overlay-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:49.630061 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.629892    4478 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a1759600-1d1e-46a2-b37b-ed2187c5df0f-provider-credentials\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:49.916159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.916079    4478 generic.go:356] "Generic (PLEG): container finished" podID="a1759600-1d1e-46a2-b37b-ed2187c5df0f" containerID="131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558" exitCode=0
Oct 31 01:11:49.916159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.916117    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"
Oct 31 01:11:49.916159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.916143    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" event={"ID":"a1759600-1d1e-46a2-b37b-ed2187c5df0f","Type":"ContainerDied","Data":"131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558"}
Oct 31 01:11:49.916403 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.916188    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc" event={"ID":"a1759600-1d1e-46a2-b37b-ed2187c5df0f","Type":"ContainerDied","Data":"64ccdb78b4cd10e5cbf82a759ab1ec73882ccecfbb2d978cd5d7ef12e8d6a3b3"}
Oct 31 01:11:49.916403 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.916209    4478 scope.go:122] "RemoveContainer" containerID="131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558"
Oct 31 01:11:49.925074 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.925056    4478 scope.go:122] "RemoveContainer" containerID="131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558"
Oct 31 01:11:49.925335 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:49.925314    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558\": container with ID starting with 131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558 not found: ID does not exist" containerID="131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558"
Oct 31 01:11:49.925393 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:49.925346    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558"} err="failed to get container status \"131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558\": rpc error: code = NotFound desc = could not find container \"131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558\": container with ID starting with 131c9873b0fba7630ab853b4974bb9ec7e0fadc4275fd5ab770b84ef31640558 not found: ID does not exist"
Oct 31 01:11:50.080042 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.080008    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"]
Oct 31 01:11:50.129182 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.129147    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-3853b6/payload-processing-external-model-e2e-extmod-a-3853b6-7ddcrlsbc"]
Oct 31 01:11:50.186169 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.186133    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerName="main" containerID="cri-o://c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4" gracePeriod=45
Oct 31 01:11:50.458938 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.458910    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 01:11:50.538624 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.538596    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf\" (UniqueName: \"kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf\") pod \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") "
Oct 31 01:11:50.538796 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.538684    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm\") pod \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") "
Oct 31 01:11:50.538796 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.538727    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home\") pod \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") "
Oct 31 01:11:50.538796 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.538749    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache\") pod \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") "
Oct 31 01:11:50.538796 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.538782    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir\") pod \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") "
Oct 31 01:11:50.539022 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.538802    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs\") pod \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\" (UID: \"c5f8009e-242f-45b3-a85e-c7faf4550aa4\") "
Oct 31 01:11:50.539085 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.539018    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home" pod "c5f8009e-242f-45b3-a85e-c7faf4550aa4" (UID: "c5f8009e-242f-45b3-a85e-c7faf4550aa4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:50.539085 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.539043    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir" pod "c5f8009e-242f-45b3-a85e-c7faf4550aa4" (UID: "c5f8009e-242f-45b3-a85e-c7faf4550aa4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:50.539085 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.539047    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache" pod "c5f8009e-242f-45b3-a85e-c7faf4550aa4" (UID: "c5f8009e-242f-45b3-a85e-c7faf4550aa4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:50.539252 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.539141    4478 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-home\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:50.539252 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.539159    4478 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-model-cache\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:50.539252 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.539172    4478 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tmp-dir\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:50.541020 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.540994    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm" pod "c5f8009e-242f-45b3-a85e-c7faf4550aa4" (UID: "c5f8009e-242f-45b3-a85e-c7faf4550aa4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 31 01:11:50.541134 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.541054    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf" pod "c5f8009e-242f-45b3-a85e-c7faf4550aa4" (UID: "c5f8009e-242f-45b3-a85e-c7faf4550aa4"). InnerVolumeSpecName "kube-api-access-fkhnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:50.541134 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.541113    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs" pod "c5f8009e-242f-45b3-a85e-c7faf4550aa4" (UID: "c5f8009e-242f-45b3-a85e-c7faf4550aa4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:11:50.640400 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.640351    4478 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5f8009e-242f-45b3-a85e-c7faf4550aa4-tls-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:50.640400 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.640407    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkhnf\" (UniqueName: \"kubernetes.io/projected/c5f8009e-242f-45b3-a85e-c7faf4550aa4-kube-api-access-fkhnf\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:50.640623 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.640420    4478 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5f8009e-242f-45b3-a85e-c7faf4550aa4-dshm\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:50.922777 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.922686    4478 generic.go:356] "Generic (PLEG): container finished" podID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerID="c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4" exitCode=0
Oct 31 01:11:50.922777 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.922738    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"
Oct 31 01:11:50.923004 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.922772    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" event={"ID":"c5f8009e-242f-45b3-a85e-c7faf4550aa4","Type":"ContainerDied","Data":"c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4"}
Oct 31 01:11:50.923004 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.922816    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" event={"ID":"c5f8009e-242f-45b3-a85e-c7faf4550aa4","Type":"ContainerDied","Data":"b1352008e7722b5bf35d0fa66fe13b20def5d4fb06fa433a1e1cf49e689c53c3"}
Oct 31 01:11:50.923004 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.922834    4478 scope.go:122] "RemoveContainer" containerID="c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4"
Oct 31 01:11:50.932088 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.932069    4478 scope.go:122] "RemoveContainer" containerID="c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4"
Oct 31 01:11:50.932346 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:50.932329    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4\": container with ID starting with c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4 not found: ID does not exist" containerID="c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4"
Oct 31 01:11:50.932423 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:50.932354    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4"} err="failed to get container status \"c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4\": rpc error: code = NotFound desc = could not find container \"c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4\": container with ID starting with c3d552716d88e622a3f01d8a76e07ba56f3347099f07ea9ff44ef4bcecd0e9c4 not found: ID does not exist"
Oct 31 01:11:51.090672 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:51.090642    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"]
Oct 31 01:11:51.094977 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:51.094957    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7"]
Oct 31 01:11:51.116641 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:51.116620    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1759600-1d1e-46a2-b37b-ed2187c5df0f" path="/var/lib/kubelet/pods/a1759600-1d1e-46a2-b37b-ed2187c5df0f/volumes"
Oct 31 01:11:51.116995 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:51.116983    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" path="/var/lib/kubelet/pods/c5f8009e-242f-45b3-a85e-c7faf4550aa4/volumes"
Oct 31 01:11:51.363772 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:51.363732    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-10a7aa/distinct-2-w0-10a7aa-kserve-6f5cfffd89-7mft7" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 31 01:11:53.021841 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.021805    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9" podUID="eab15438-e78d-42b5-ad77-8ec01199555d" containerName="payload-processing" containerID="cri-o://df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5" gracePeriod=25
Oct 31 01:11:53.275132 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.275071    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:11:53.366053 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366026    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume\") pod \"eab15438-e78d-42b5-ad77-8ec01199555d\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") "
Oct 31 01:11:53.366224 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366086    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk\") pod \"eab15438-e78d-42b5-ad77-8ec01199555d\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") "
Oct 31 01:11:53.366296 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366264    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume\") pod \"eab15438-e78d-42b5-ad77-8ec01199555d\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") "
Oct 31 01:11:53.366360 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366309    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials\") pod \"eab15438-e78d-42b5-ad77-8ec01199555d\" (UID: \"eab15438-e78d-42b5-ad77-8ec01199555d\") "
Oct 31 01:11:53.366470 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366448    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume" pod "eab15438-e78d-42b5-ad77-8ec01199555d" (UID: "eab15438-e78d-42b5-ad77-8ec01199555d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:53.366671 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366652    4478 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-routing-overlay-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:53.366731 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.366681    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume" pod "eab15438-e78d-42b5-ad77-8ec01199555d" (UID: "eab15438-e78d-42b5-ad77-8ec01199555d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:11:53.368470 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.368449    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk" pod "eab15438-e78d-42b5-ad77-8ec01199555d" (UID: "eab15438-e78d-42b5-ad77-8ec01199555d"). InnerVolumeSpecName "kube-api-access-24gwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:53.368564 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.368546    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials" pod "eab15438-e78d-42b5-ad77-8ec01199555d" (UID: "eab15438-e78d-42b5-ad77-8ec01199555d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:11:53.467303 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.467271    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24gwk\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-kube-api-access-24gwk\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:53.467303 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.467300    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eab15438-e78d-42b5-ad77-8ec01199555d-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:53.467514 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.467310    4478 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/eab15438-e78d-42b5-ad77-8ec01199555d-provider-credentials\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:11:53.940417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.940362    4478 generic.go:356] "Generic (PLEG): container finished" podID="eab15438-e78d-42b5-ad77-8ec01199555d" containerID="df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5" exitCode=0
Oct 31 01:11:53.940417 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.940428    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"
Oct 31 01:11:53.940693 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.940454    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9" event={"ID":"eab15438-e78d-42b5-ad77-8ec01199555d","Type":"ContainerDied","Data":"df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5"}
Oct 31 01:11:53.940693 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.940523    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9" event={"ID":"eab15438-e78d-42b5-ad77-8ec01199555d","Type":"ContainerDied","Data":"1c2aa34c50338f5dbf1fa9fd3651e28d36db9268c8446969b72520de004374f4"}
Oct 31 01:11:53.940693 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.940553    4478 scope.go:122] "RemoveContainer" containerID="df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5"
Oct 31 01:11:53.949297 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.949281    4478 scope.go:122] "RemoveContainer" containerID="df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5"
Oct 31 01:11:53.949587 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:11:53.949568    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5\": container with ID starting with df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5 not found: ID does not exist" containerID="df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5"
Oct 31 01:11:53.949652 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:53.949593    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5"} err="failed to get container status \"df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5\": rpc error: code = NotFound desc = could not find container \"df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5\": container with ID starting with df0300f4ebb7e31d761af3ea218581e4c701c01eed8daa3a699e3ebff26061d5 not found: ID does not exist"
Oct 31 01:11:54.103336 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:54.103311    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"]
Oct 31 01:11:54.152783 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:54.152754    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667dxp7x9"]
Oct 31 01:11:55.116450 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:11:55.116417    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab15438-e78d-42b5-ad77-8ec01199555d" path="/var/lib/kubelet/pods/eab15438-e78d-42b5-ad77-8ec01199555d/volumes"
Oct 31 01:12:14.756321 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:14.756286    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:12:38.756407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:38.756317    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"]
Oct 31 01:12:38.756842 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:38.756603    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" podUID="629634a5-2a44-4c5a-8f76-9e6c29c960c3" containerName="maas-api" containerID="cri-o://33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6" gracePeriod=30
Oct 31 01:12:39.016038 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.015987    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 01:12:39.089517 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.089488    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca\") pod \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") "
Oct 31 01:12:39.089656 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.089558    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn\" (UniqueName: \"kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn\") pod \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") "
Oct 31 01:12:39.089711 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.089659    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs\") pod \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") "
Oct 31 01:12:39.089811 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.089794    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\") pod \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\" (UID: \"629634a5-2a44-4c5a-8f76-9e6c29c960c3\") "
Oct 31 01:12:39.089811 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.089803    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca" pod "629634a5-2a44-4c5a-8f76-9e6c29c960c3" (UID: "629634a5-2a44-4c5a-8f76-9e6c29c960c3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:12:39.090089 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.090072    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/629634a5-2a44-4c5a-8f76-9e6c29c960c3-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:12:39.091865 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.091843    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn" pod "629634a5-2a44-4c5a-8f76-9e6c29c960c3" (UID: "629634a5-2a44-4c5a-8f76-9e6c29c960c3"). InnerVolumeSpecName "kube-api-access-qfxvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:12:39.091865 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.091857    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls" pod "629634a5-2a44-4c5a-8f76-9e6c29c960c3" (UID: "629634a5-2a44-4c5a-8f76-9e6c29c960c3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:12:39.091865 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.091842    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs" pod "629634a5-2a44-4c5a-8f76-9e6c29c960c3" (UID: "629634a5-2a44-4c5a-8f76-9e6c29c960c3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:12:39.164576 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.164544    4478 generic.go:356] "Generic (PLEG): container finished" podID="629634a5-2a44-4c5a-8f76-9e6c29c960c3" containerID="33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6" exitCode=0
Oct 31 01:12:39.164702 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.164585    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"
Oct 31 01:12:39.164702 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.164615    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" event={"ID":"629634a5-2a44-4c5a-8f76-9e6c29c960c3","Type":"ContainerDied","Data":"33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6"}
Oct 31 01:12:39.164702 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.164663    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4" event={"ID":"629634a5-2a44-4c5a-8f76-9e6c29c960c3","Type":"ContainerDied","Data":"249e13a2878d5e475e380a037ee981c801364b506139cf9d2f9ffe2a7ecf984f"}
Oct 31 01:12:39.164702 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.164686    4478 scope.go:122] "RemoveContainer" containerID="33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6"
Oct 31 01:12:39.173346 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.173327    4478 scope.go:122] "RemoveContainer" containerID="33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6"
Oct 31 01:12:39.173657 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:39.173640    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6\": container with ID starting with 33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6 not found: ID does not exist" containerID="33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6"
Oct 31 01:12:39.173715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.173665    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6"} err="failed to get container status \"33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6\": rpc error: code = NotFound desc = could not find container \"33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6\": container with ID starting with 33ad43ba352d48288babaff8cd344746d51f7bd570d440cc52c136752a47edc6 not found: ID does not exist"
Oct 31 01:12:39.191249 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.191225    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:12:39.191373 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.191252    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfxvn\" (UniqueName: \"kubernetes.io/projected/629634a5-2a44-4c5a-8f76-9e6c29c960c3-kube-api-access-qfxvn\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:12:39.191373 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.191268    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/629634a5-2a44-4c5a-8f76-9e6c29c960c3-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:12:39.329568 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.329490    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"]
Oct 31 01:12:39.334647 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:39.334624    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-10a7aa-6c6fb96c69-xpcv4"]
Oct 31 01:12:41.116061 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:41.116029    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629634a5-2a44-4c5a-8f76-9e6c29c960c3" path="/var/lib/kubelet/pods/629634a5-2a44-4c5a-8f76-9e6c29c960c3/volumes"
Oct 31 01:12:44.757343 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:44.757306    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:12:52.788159 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788120    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"]
Oct 31 01:12:52.788667 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788650    4478 cpu_manager.go:414] "containerMap: removing container" podUID="eab15438-e78d-42b5-ad77-8ec01199555d" containerName="payload-processing"
Oct 31 01:12:52.788715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788670    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab15438-e78d-42b5-ad77-8ec01199555d" containerName="payload-processing"
Oct 31 01:12:52.788715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788680    4478 cpu_manager.go:414] "containerMap: removing container" podUID="a1759600-1d1e-46a2-b37b-ed2187c5df0f" containerName="payload-processing"
Oct 31 01:12:52.788715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788687    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1759600-1d1e-46a2-b37b-ed2187c5df0f" containerName="payload-processing"
Oct 31 01:12:52.788715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788702    4478 cpu_manager.go:414] "containerMap: removing container" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerName="main"
Oct 31 01:12:52.788715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788708    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerName="main"
Oct 31 01:12:52.788715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788716    4478 cpu_manager.go:414] "containerMap: removing container" podUID="629634a5-2a44-4c5a-8f76-9e6c29c960c3" containerName="maas-api"
Oct 31 01:12:52.788898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788721    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629634a5-2a44-4c5a-8f76-9e6c29c960c3" containerName="maas-api"
Oct 31 01:12:52.788898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788793    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f8009e-242f-45b3-a85e-c7faf4550aa4" containerName="main"
Oct 31 01:12:52.788898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788804    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="629634a5-2a44-4c5a-8f76-9e6c29c960c3" containerName="maas-api"
Oct 31 01:12:52.788898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788817    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab15438-e78d-42b5-ad77-8ec01199555d" containerName="payload-processing"
Oct 31 01:12:52.788898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.788825    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1759600-1d1e-46a2-b37b-ed2187c5df0f" containerName="payload-processing"
Oct 31 01:12:52.793374 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.793358    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:52.848649 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.848621    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"]
Oct 31 01:12:52.919475 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.919440    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:52.919645 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.919610    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzs5k\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:52.919710 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.919651    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:52.919710 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:52.919685    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.021420 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.021357    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.021713 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.021691    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzs5k\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.021797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.021730    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.021797 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.021765    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.022587 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.022561    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.022689 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.022627    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.024570 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.024545    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.083526 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.083453    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzs5k\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k\") pod \"payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") " pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.107632 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.107606    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:53.481400 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:53.481350    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"]
Oct 31 01:12:53.484557 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:12:53.484529    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33ea6c3c_609c_4df9_b2ee_c2027372e2c3.slice/crio-cd43efee8026327e50a3a58e8af20408b30ede4305a9989e93a02cc0a20305d5 WatchSource:0}: Error finding container cd43efee8026327e50a3a58e8af20408b30ede4305a9989e93a02cc0a20305d5: Status 404 returned error can't find the container with id cd43efee8026327e50a3a58e8af20408b30ede4305a9989e93a02cc0a20305d5
Oct 31 01:12:54.243174 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:54.243132    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" event={"ID":"33ea6c3c-609c-4df9-b2ee-c2027372e2c3","Type":"ContainerStarted","Data":"b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b"}
Oct 31 01:12:54.243701 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:54.243194    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" event={"ID":"33ea6c3c-609c-4df9-b2ee-c2027372e2c3","Type":"ContainerStarted","Data":"cd43efee8026327e50a3a58e8af20408b30ede4305a9989e93a02cc0a20305d5"}
Oct 31 01:12:54.243701 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:54.243349    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:12:54.360693 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:12:54.360642    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" podStartSLOduration=2.360625792 podStartE2EDuration="2.360625792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 01:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:12:54.358635961 +0000 UTC m=+4207.375300703" watchObservedRunningTime="2026-10-31 01:12:54.360625792 +0000 UTC m=+4207.377290503"
Oct 31 01:12:58.271541 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:58.271507    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d7412d/agc-cred-89de3a35c154: secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:58.271541 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:58.271535    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4: secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:58.272048 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:58.271607    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials podName:13e1e103-4888-46a9-87c1-b73ae7cad0d1 nodeName:}" failed. No retries permitted until 2026-10-31 01:12:58.77158511 +0000 UTC m=+4211.788249823 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1") : secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:58.777164 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:58.777133    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d7412d/agc-cred-89de3a35c154: secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:58.777164 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:58.777156    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4: secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:58.777404 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:58.777208    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials podName:13e1e103-4888-46a9-87c1-b73ae7cad0d1 nodeName:}" failed. No retries permitted until 2026-10-31 01:12:59.777190855 +0000 UTC m=+4212.793855548 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1") : secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:59.788436 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:59.788376    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d7412d/agc-cred-89de3a35c154: secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:59.788436 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:59.788425    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4: secret "agc-cred-89de3a35c154" not found
Oct 31 01:12:59.788963 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:12:59.788482    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials podName:13e1e103-4888-46a9-87c1-b73ae7cad0d1 nodeName:}" failed. No retries permitted until 2026-10-31 01:13:01.788465882 +0000 UTC m=+4214.805130588 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1") : secret "agc-cred-89de3a35c154" not found
Oct 31 01:13:01.807517 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:01.807487    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d7412d/agc-cred-89de3a35c154: secret "agc-cred-89de3a35c154" not found
Oct 31 01:13:01.807517 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:01.807513    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4: secret "agc-cred-89de3a35c154" not found
Oct 31 01:13:01.807945 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:01.807566    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials podName:13e1e103-4888-46a9-87c1-b73ae7cad0d1 nodeName:}" failed. No retries permitted until 2026-10-31 01:13:05.807551215 +0000 UTC m=+4218.824215906 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1") : secret "agc-cred-89de3a35c154" not found
Oct 31 01:13:04.867258 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:04.867222    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"]
Oct 31 01:13:04.916392 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:04.916348    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"]
Oct 31 01:13:04.918840 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:04.918812    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" podUID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:13:05.848803 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:05.848769    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d7412d/agc-cred-2e3729eb8072: secret "agc-cred-2e3729eb8072" not found
Oct 31 01:13:05.848803 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:05.848796    4478 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-b-d7412d/agc-cred-89de3a35c154: secret "agc-cred-89de3a35c154" not found
Oct 31 01:13:05.848803 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:05.848809    4478 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4: [secret "agc-cred-2e3729eb8072" not found, secret "agc-cred-89de3a35c154" not found]
Oct 31 01:13:05.849110 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:05.848875    4478 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials podName:13e1e103-4888-46a9-87c1-b73ae7cad0d1 nodeName:}" failed. No retries permitted until 2026-10-31 01:13:13.848854876 +0000 UTC m=+4226.865519581 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials") pod "payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1") : [secret "agc-cred-2e3729eb8072" not found, secret "agc-cred-89de3a35c154" not found]
Oct 31 01:13:09.867559 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:09.867517    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" podUID="13e1e103-4888-46a9-87c1-b73ae7cad0d1" containerName="payload-processing" containerID="cri-o://1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3" gracePeriod=25
Oct 31 01:13:09.917167 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:09.917114    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" podUID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" containerName="payload-processing" containerID="cri-o://b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b" gracePeriod=25
Oct 31 01:13:10.183759 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.183731    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:13:10.186522 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.186500    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:13:10.291799 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.291768    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials\") pod \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") "
Oct 31 01:13:10.291965 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.291819    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume\") pod \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") "
Oct 31 01:13:10.291965 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.291851    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume\") pod \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") "
Oct 31 01:13:10.291965 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.291895    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials\") pod \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") "
Oct 31 01:13:10.292162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292009    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume\") pod \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") "
Oct 31 01:13:10.292162 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292061    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k\") pod \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\" (UID: \"33ea6c3c-609c-4df9-b2ee-c2027372e2c3\") "
Oct 31 01:13:10.292282 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292165    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume\") pod \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") "
Oct 31 01:13:10.292282 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292199    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk\") pod \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\" (UID: \"13e1e103-4888-46a9-87c1-b73ae7cad0d1\") "
Oct 31 01:13:10.292680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292195    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume" pod "33ea6c3c-609c-4df9-b2ee-c2027372e2c3" (UID: "33ea6c3c-609c-4df9-b2ee-c2027372e2c3"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:10.292680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292636    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume" pod "13e1e103-4888-46a9-87c1-b73ae7cad0d1" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:10.292680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292351    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume" pod "13e1e103-4888-46a9-87c1-b73ae7cad0d1" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:10.292680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.292504    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume" pod "33ea6c3c-609c-4df9-b2ee-c2027372e2c3" (UID: "33ea6c3c-609c-4df9-b2ee-c2027372e2c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:13:10.293357 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.293088    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.293357 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.293112    4478 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-routing-overlay-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.293357 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.293127    4478 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/13e1e103-4888-46a9-87c1-b73ae7cad0d1-routing-overlay-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.293357 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.293145    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.294637 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.294612    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials" pod "13e1e103-4888-46a9-87c1-b73ae7cad0d1" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:10.294732 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.294648    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk" pod "13e1e103-4888-46a9-87c1-b73ae7cad0d1" (UID: "13e1e103-4888-46a9-87c1-b73ae7cad0d1"). InnerVolumeSpecName "kube-api-access-4wswk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:10.294788 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.294742    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials" pod "33ea6c3c-609c-4df9-b2ee-c2027372e2c3" (UID: "33ea6c3c-609c-4df9-b2ee-c2027372e2c3"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:10.295057 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.295042    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k" pod "33ea6c3c-609c-4df9-b2ee-c2027372e2c3" (UID: "33ea6c3c-609c-4df9-b2ee-c2027372e2c3"). InnerVolumeSpecName "kube-api-access-gzs5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:13:10.319570 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.319538    4478 generic.go:356] "Generic (PLEG): container finished" podID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" containerID="b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b" exitCode=0
Oct 31 01:13:10.319725 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.319596    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"
Oct 31 01:13:10.319725 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.319622    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" event={"ID":"33ea6c3c-609c-4df9-b2ee-c2027372e2c3","Type":"ContainerDied","Data":"b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b"}
Oct 31 01:13:10.319725 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.319676    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg" event={"ID":"33ea6c3c-609c-4df9-b2ee-c2027372e2c3","Type":"ContainerDied","Data":"cd43efee8026327e50a3a58e8af20408b30ede4305a9989e93a02cc0a20305d5"}
Oct 31 01:13:10.319725 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.319704    4478 scope.go:122] "RemoveContainer" containerID="b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b"
Oct 31 01:13:10.321012 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.320988    4478 generic.go:356] "Generic (PLEG): container finished" podID="13e1e103-4888-46a9-87c1-b73ae7cad0d1" containerID="1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3" exitCode=0
Oct 31 01:13:10.321137 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.321015    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" event={"ID":"13e1e103-4888-46a9-87c1-b73ae7cad0d1","Type":"ContainerDied","Data":"1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3"}
Oct 31 01:13:10.321137 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.321036    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"
Oct 31 01:13:10.321137 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.321061    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4" event={"ID":"13e1e103-4888-46a9-87c1-b73ae7cad0d1","Type":"ContainerDied","Data":"9457f5a12e23367df5c50f49780253264d20fa846698bc6dc97baaf47748c674"}
Oct 31 01:13:10.330788 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.330771    4478 scope.go:122] "RemoveContainer" containerID="b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b"
Oct 31 01:13:10.331076 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:10.331057    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b\": container with ID starting with b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b not found: ID does not exist" containerID="b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b"
Oct 31 01:13:10.331148 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.331083    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b"} err="failed to get container status \"b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b\": rpc error: code = NotFound desc = could not find container \"b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b\": container with ID starting with b8ce352efaeec70d46ae484816129137ff438ca80f2e014506020c75517c794b not found: ID does not exist"
Oct 31 01:13:10.331148 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.331100    4478 scope.go:122] "RemoveContainer" containerID="1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3"
Oct 31 01:13:10.339303 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.339284    4478 scope.go:122] "RemoveContainer" containerID="1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3"
Oct 31 01:13:10.339612 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:10.339595    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3\": container with ID starting with 1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3 not found: ID does not exist" containerID="1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3"
Oct 31 01:13:10.339673 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.339619    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3"} err="failed to get container status \"1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3\": rpc error: code = NotFound desc = could not find container \"1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3\": container with ID starting with 1fcb53bdef39d017dbb107e2f49e39440376ead6320886751c93c88437c08fc3 not found: ID does not exist"
Oct 31 01:13:10.393848 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.393818    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzs5k\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-kube-api-access-gzs5k\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.393848 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.393847    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wswk\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-kube-api-access-4wswk\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.394030 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.393861    4478 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/13e1e103-4888-46a9-87c1-b73ae7cad0d1-provider-credentials\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.394030 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.393873    4478 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33ea6c3c-609c-4df9-b2ee-c2027372e2c3-provider-credentials\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:13:10.528132 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.528099    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"]
Oct 31 01:13:10.531864 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.531832    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-7ccchwwh4"]
Oct 31 01:13:10.639874 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.639843    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"]
Oct 31 01:13:10.645256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:10.645230    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d7412d/payload-processing-external-model-e2e-extmod-b-d7412d-667d8zqxg"]
Oct 31 01:13:11.116515 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:11.116485    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13e1e103-4888-46a9-87c1-b73ae7cad0d1" path="/var/lib/kubelet/pods/13e1e103-4888-46a9-87c1-b73ae7cad0d1/volumes"
Oct 31 01:13:11.116894 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:13:11.116840    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" path="/var/lib/kubelet/pods/33ea6c3c-609c-4df9-b2ee-c2027372e2c3/volumes"
Oct 31 01:13:14.759187 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:14.759150    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:13:44.757483 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:13:44.757448    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:14:14.756473 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:14:14.756331    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:14:30.322889 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:30.322854    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"]
Oct 31 01:14:30.426184 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:30.426139    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"]
Oct 31 01:14:35.323511 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:35.323426    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerName="payload-processing" containerID="cri-o://7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea" gracePeriod=25
Oct 31 01:14:35.427362 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:35.427307    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" podUID="1689d832-f216-4ebc-aa4e-650155a07944" containerName="payload-pre-processing" containerID="cri-o://52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a" gracePeriod=25
Oct 31 01:14:38.130919 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:38.130879    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" podUID="1689d832-f216-4ebc-aa4e-650155a07944" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:14:38.131305 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:38.130893    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:14:44.756301 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:14:44.756265    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:14:48.130958 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:48.130919    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" podUID="1689d832-f216-4ebc-aa4e-650155a07944" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:14:48.131407 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:48.130987    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:14:55.575095 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.575070    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 01:14:55.648332 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.648300    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume\") pod \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") "
Oct 31 01:14:55.648486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.648341    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6\" (UniqueName: \"kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6\") pod \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\" (UID: \"07ef74ae-1614-4a6c-aa76-a9761c27d98c\") "
Oct 31 01:14:55.648812 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.648787    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume" pod "07ef74ae-1614-4a6c-aa76-a9761c27d98c" (UID: "07ef74ae-1614-4a6c-aa76-a9761c27d98c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:55.650891 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.650862    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6" pod "07ef74ae-1614-4a6c-aa76-a9761c27d98c" (UID: "07ef74ae-1614-4a6c-aa76-a9761c27d98c"). InnerVolumeSpecName "kube-api-access-kh7n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:55.660149 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.660127    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 01:14:55.749280 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.749252    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f\" (UniqueName: \"kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f\") pod \"1689d832-f216-4ebc-aa4e-650155a07944\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") "
Oct 31 01:14:55.749454 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.749419    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume\") pod \"1689d832-f216-4ebc-aa4e-650155a07944\" (UID: \"1689d832-f216-4ebc-aa4e-650155a07944\") "
Oct 31 01:14:55.749725 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.749707    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07ef74ae-1614-4a6c-aa76-a9761c27d98c-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:14:55.749793 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.749726    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh7n6\" (UniqueName: \"kubernetes.io/projected/07ef74ae-1614-4a6c-aa76-a9761c27d98c-kube-api-access-kh7n6\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:14:55.749793 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.749728    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume" pod "1689d832-f216-4ebc-aa4e-650155a07944" (UID: "1689d832-f216-4ebc-aa4e-650155a07944"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:14:55.751623 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.751605    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f" pod "1689d832-f216-4ebc-aa4e-650155a07944" (UID: "1689d832-f216-4ebc-aa4e-650155a07944"). InnerVolumeSpecName "kube-api-access-k4b8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:14:55.797822 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.797793    4478 generic.go:356] "Generic (PLEG): container finished" podID="1689d832-f216-4ebc-aa4e-650155a07944" containerID="52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a" exitCode=0
Oct 31 01:14:55.797935 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.797827    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" event={"ID":"1689d832-f216-4ebc-aa4e-650155a07944","Type":"ContainerDied","Data":"52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a"}
Oct 31 01:14:55.797935 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.797835    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"
Oct 31 01:14:55.797935 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.797888    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp" event={"ID":"1689d832-f216-4ebc-aa4e-650155a07944","Type":"ContainerDied","Data":"1cc6f15f1da12c32accad54e694f5f7a35e111d47e67648c30bebe4355bd6e4b"}
Oct 31 01:14:55.797935 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.797911    4478 scope.go:122] "RemoveContainer" containerID="52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a"
Oct 31 01:14:55.799290 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.799268    4478 generic.go:356] "Generic (PLEG): container finished" podID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerID="7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea" exitCode=0
Oct 31 01:14:55.799406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.799298    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" event={"ID":"07ef74ae-1614-4a6c-aa76-a9761c27d98c","Type":"ContainerDied","Data":"7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea"}
Oct 31 01:14:55.799406 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.799348    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l" event={"ID":"07ef74ae-1614-4a6c-aa76-a9761c27d98c","Type":"ContainerDied","Data":"8304b236c469d7a8c914f04915822d535ce63931ba99ff9f1c82ce53b89517ff"}
Oct 31 01:14:55.799496 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.799404    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"
Oct 31 01:14:55.809125 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.809040    4478 scope.go:122] "RemoveContainer" containerID="52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a"
Oct 31 01:14:55.809545 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:14:55.809519    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a\": container with ID starting with 52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a not found: ID does not exist" containerID="52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a"
Oct 31 01:14:55.810501 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.809553    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a"} err="failed to get container status \"52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a\": rpc error: code = NotFound desc = could not find container \"52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a\": container with ID starting with 52c16c239db03fe8725805e7ca73d4e491c06521401e40dd4add19f81380681a not found: ID does not exist"
Oct 31 01:14:55.810501 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.809605    4478 scope.go:122] "RemoveContainer" containerID="7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea"
Oct 31 01:14:55.819689 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.819667    4478 scope.go:122] "RemoveContainer" containerID="7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea"
Oct 31 01:14:55.820006 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:14:55.819982    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea\": container with ID starting with 7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea not found: ID does not exist" containerID="7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea"
Oct 31 01:14:55.820096 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.820016    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea"} err="failed to get container status \"7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea\": rpc error: code = NotFound desc = could not find container \"7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea\": container with ID starting with 7b0af88da194c3861af47ff2e3ce584c1da068fc120340106a3a74dddd192dea not found: ID does not exist"
Oct 31 01:14:55.850355 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.850307    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1689d832-f216-4ebc-aa4e-650155a07944-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:14:55.850355 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:55.850328    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4b8f\" (UniqueName: \"kubernetes.io/projected/1689d832-f216-4ebc-aa4e-650155a07944-kube-api-access-k4b8f\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:14:56.021125 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:56.021098    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"]
Oct 31 01:14:56.026467 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:56.026444    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-4743f8-5b6b746d7f-j7gmp"]
Oct 31 01:14:56.135498 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:56.135423    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"]
Oct 31 01:14:56.138898 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:56.138869    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-4743f8-7bf4457cd9-kt46l"]
Oct 31 01:14:57.116226 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:57.116186    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" path="/var/lib/kubelet/pods/07ef74ae-1614-4a6c-aa76-a9761c27d98c/volumes"
Oct 31 01:14:57.116615 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:14:57.116584    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1689d832-f216-4ebc-aa4e-650155a07944" path="/var/lib/kubelet/pods/1689d832-f216-4ebc-aa4e-650155a07944/volumes"
Oct 31 01:15:05.661486 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:05.661444    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"]
Oct 31 01:15:10.662367 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:10.662322    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerName="payload-pre-processing" containerID="cri-o://e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f" gracePeriod=25
Oct 31 01:15:14.086993 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:14.086953    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:15:14.666608 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:14.666570    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"]
Oct 31 01:15:14.666830 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:14.666791    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" podUID="695c4a13-794d-42da-a8ed-a88e0a19610e" containerName="maas-api" containerID="cri-o://42da25b5dce2b9a4be24a3cf47d3815f5113980153cce0c4a90dfd796dbc01a9" gracePeriod=30
Oct 31 01:15:14.762260 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:15:14.762220    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:15:14.899264 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:14.899225    4478 generic.go:356] "Generic (PLEG): container finished" podID="695c4a13-794d-42da-a8ed-a88e0a19610e" containerID="42da25b5dce2b9a4be24a3cf47d3815f5113980153cce0c4a90dfd796dbc01a9" exitCode=0
Oct 31 01:15:14.899421 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:14.899289    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" event={"ID":"695c4a13-794d-42da-a8ed-a88e0a19610e","Type":"ContainerDied","Data":"42da25b5dce2b9a4be24a3cf47d3815f5113980153cce0c4a90dfd796dbc01a9"}
Oct 31 01:15:14.916746 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:14.916687    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 01:15:15.016868 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.016845    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt\" (UniqueName: \"kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt\") pod \"695c4a13-794d-42da-a8ed-a88e0a19610e\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") "
Oct 31 01:15:15.017001 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.016882    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca\") pod \"695c4a13-794d-42da-a8ed-a88e0a19610e\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") "
Oct 31 01:15:15.017001 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.016918    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs\") pod \"695c4a13-794d-42da-a8ed-a88e0a19610e\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") "
Oct 31 01:15:15.017087 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.017001    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\") pod \"695c4a13-794d-42da-a8ed-a88e0a19610e\" (UID: \"695c4a13-794d-42da-a8ed-a88e0a19610e\") "
Oct 31 01:15:15.017273 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.017250    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca" pod "695c4a13-794d-42da-a8ed-a88e0a19610e" (UID: "695c4a13-794d-42da-a8ed-a88e0a19610e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:15.019104 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.019081    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt" pod "695c4a13-794d-42da-a8ed-a88e0a19610e" (UID: "695c4a13-794d-42da-a8ed-a88e0a19610e"). InnerVolumeSpecName "kube-api-access-z66gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:15.019244 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.019224    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls" pod "695c4a13-794d-42da-a8ed-a88e0a19610e" (UID: "695c4a13-794d-42da-a8ed-a88e0a19610e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:15.019299 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.019241    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs" pod "695c4a13-794d-42da-a8ed-a88e0a19610e" (UID: "695c4a13-794d-42da-a8ed-a88e0a19610e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:15.117715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.117683    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:15.117715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.117711    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z66gt\" (UniqueName: \"kubernetes.io/projected/695c4a13-794d-42da-a8ed-a88e0a19610e-kube-api-access-z66gt\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:15.117715 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.117722    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/695c4a13-794d-42da-a8ed-a88e0a19610e-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:15.118323 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.117730    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/695c4a13-794d-42da-a8ed-a88e0a19610e-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:15.904275 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.904245    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"
Oct 31 01:15:15.904498 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.904249    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs" event={"ID":"695c4a13-794d-42da-a8ed-a88e0a19610e","Type":"ContainerDied","Data":"6fb6c85f4451a89c66ad567ff3af610c56815daf16a7f95a1785c1307b69b741"}
Oct 31 01:15:15.904498 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:15.904391    4478 scope.go:122] "RemoveContainer" containerID="42da25b5dce2b9a4be24a3cf47d3815f5113980153cce0c4a90dfd796dbc01a9"
Oct 31 01:15:16.067421 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:16.067369    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"]
Oct 31 01:15:16.116697 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:16.116666    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-093220-8fd4c6d6b-b5nzs"]
Oct 31 01:15:17.116598 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:17.116566    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="695c4a13-794d-42da-a8ed-a88e0a19610e" path="/var/lib/kubelet/pods/695c4a13-794d-42da-a8ed-a88e0a19610e/volumes"
Oct 31 01:15:24.087136 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:24.087094    4478 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 31 01:15:30.908628 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.908603    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 01:15:30.950969 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.950934    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r\" (UniqueName: \"kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r\") pod \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") "
Oct 31 01:15:30.951168 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.951092    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume\") pod \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\" (UID: \"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9\") "
Oct 31 01:15:30.951472 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.951449    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume" pod "fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" (UID: "fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:30.953217 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.953181    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r" pod "fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" (UID: "fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9"). InnerVolumeSpecName "kube-api-access-shm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:30.971094 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.971069    4478 generic.go:356] "Generic (PLEG): container finished" podID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerID="e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f" exitCode=0
Oct 31 01:15:30.971230 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.971170    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"
Oct 31 01:15:30.971230 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.971214    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" event={"ID":"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9","Type":"ContainerDied","Data":"e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f"}
Oct 31 01:15:30.971354 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.971280    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v" event={"ID":"fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9","Type":"ContainerDied","Data":"61ed8ef2d20782a9416edb2b2ea89ea128577da15247539f84c9dba858baff74"}
Oct 31 01:15:30.971354 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.971311    4478 scope.go:122] "RemoveContainer" containerID="e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f"
Oct 31 01:15:30.980695 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.980680    4478 scope.go:122] "RemoveContainer" containerID="e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f"
Oct 31 01:15:30.980986 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:15:30.980972    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f\": container with ID starting with e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f not found: ID does not exist" containerID="e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f"
Oct 31 01:15:30.981058 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:30.980994    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f"} err="failed to get container status \"e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f\": rpc error: code = NotFound desc = could not find container \"e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f\": container with ID starting with e4c3c89fb2d2bb20784f6b807bdfd0f3e6dab2ff91c018d832b889e87b74e54f not found: ID does not exist"
Oct 31 01:15:31.052373 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:31.052344    4478 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-plugins-config-volume\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:31.052373 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:31.052370    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shm6r\" (UniqueName: \"kubernetes.io/projected/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9-kube-api-access-shm6r\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:31.136352 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:31.136316    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"]
Oct 31 01:15:31.142329 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:31.142301    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-093220-744cc84bb7-blh8v"]
Oct 31 01:15:33.116194 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:33.116158    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" path="/var/lib/kubelet/pods/fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9/volumes"
Oct 31 01:15:44.774304 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:15:44.774254    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:15:50.465278 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.465240    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"]
Oct 31 01:15:50.465710 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.465486    4478 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" podUID="b68e8749-2183-4329-97c7-8f5c9ddb0fbb" containerName="maas-api" containerID="cri-o://01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c" gracePeriod=30
Oct 31 01:15:50.717812 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.717758    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 01:15:50.832912 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.832882    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs\") pod \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") "
Oct 31 01:15:50.833066 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.832928    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\") pod \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") "
Oct 31 01:15:50.833066 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.833030    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp\" (UniqueName: \"kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp\") pod \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") "
Oct 31 01:15:50.833191 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.833071    4478 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca\") pod \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\" (UID: \"b68e8749-2183-4329-97c7-8f5c9ddb0fbb\") "
Oct 31 01:15:50.833490 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.833462    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca" pod "b68e8749-2183-4329-97c7-8f5c9ddb0fbb" (UID: "b68e8749-2183-4329-97c7-8f5c9ddb0fbb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 31 01:15:50.835124 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.835101    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls" pod "b68e8749-2183-4329-97c7-8f5c9ddb0fbb" (UID: "b68e8749-2183-4329-97c7-8f5c9ddb0fbb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:50.835124 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.835118    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs" pod "b68e8749-2183-4329-97c7-8f5c9ddb0fbb" (UID: "b68e8749-2183-4329-97c7-8f5c9ddb0fbb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 31 01:15:50.835275 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.835149    4478 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp" pod "b68e8749-2183-4329-97c7-8f5c9ddb0fbb" (UID: "b68e8749-2183-4329-97c7-8f5c9ddb0fbb"). InnerVolumeSpecName "kube-api-access-p8mcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 31 01:15:50.934060 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.934023    4478 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8mcp\" (UniqueName: \"kubernetes.io/projected/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-kube-api-access-p8mcp\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:50.934060 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.934054    4478 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-openshift-service-ca\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:50.934060 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.934067    4478 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-metrics-certs\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:50.934303 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:50.934079    4478 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b68e8749-2183-4329-97c7-8f5c9ddb0fbb-maas-api-tls\") on node \"ip-10-0-137-148.us-west-1.compute.internal\" DevicePath \"\""
Oct 31 01:15:51.062763 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.062683    4478 generic.go:356] "Generic (PLEG): container finished" podID="b68e8749-2183-4329-97c7-8f5c9ddb0fbb" containerID="01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c" exitCode=0
Oct 31 01:15:51.062763 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.062728    4478 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"
Oct 31 01:15:51.062975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.062771    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" event={"ID":"b68e8749-2183-4329-97c7-8f5c9ddb0fbb","Type":"ContainerDied","Data":"01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c"}
Oct 31 01:15:51.062975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.062830    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp" event={"ID":"b68e8749-2183-4329-97c7-8f5c9ddb0fbb","Type":"ContainerDied","Data":"d9aac23c96a3424415eb86e5a667c0ac01d3c56045062d25e7ebb7d0793d47d2"}
Oct 31 01:15:51.062975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.062852    4478 scope.go:122] "RemoveContainer" containerID="01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c"
Oct 31 01:15:51.072712 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.072698    4478 scope.go:122] "RemoveContainer" containerID="01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c"
Oct 31 01:15:51.072967 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:15:51.072946    4478 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c\": container with ID starting with 01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c not found: ID does not exist" containerID="01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c"
Oct 31 01:15:51.073038 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.072980    4478 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c"} err="failed to get container status \"01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c\": rpc error: code = NotFound desc = could not find container \"01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c\": container with ID starting with 01af7f8fbc906fe4593b24be23ef4484d041dd75469a7c2520f35ad48cd13c0c not found: ID does not exist"
Oct 31 01:15:51.225433 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.225398    4478 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"]
Oct 31 01:15:51.229860 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:51.229838    4478 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-0144bd-5b6fd46899-mwhgp"]
Oct 31 01:15:53.115363 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:15:53.115326    4478 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b68e8749-2183-4329-97c7-8f5c9ddb0fbb" path="/var/lib/kubelet/pods/b68e8749-2183-4329-97c7-8f5c9ddb0fbb/volumes"
Oct 31 01:16:01.351175 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:01.351147    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4bb8f767-9xcl4_450dc4e8-04b1-40cb-89a7-71d635a80d9d/authorino/0.log"
Oct 31 01:16:04.090732 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:04.090692    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zbvhk_a90600f9-ec16-4ae5-af38-f60501b5ffb6/manager/0.log"
Oct 31 01:16:05.593508 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:05.593474    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:16:05.869217 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:05.869129    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:16:08.952033 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:08.952004    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4bb8f767-9xcl4_450dc4e8-04b1-40cb-89a7-71d635a80d9d/authorino/0.log"
Oct 31 01:16:14.756669 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:16:14.756637    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:16:19.769945 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:19.769908    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:16:20.142656 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:20.142572    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:16:23.277638 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:23.277611    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-78c8449567-7vh5p_ab82044d-5a0c-4638-a676-5ec869d85173/maas-api/0.log"
Oct 31 01:16:26.678318 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:26.678288    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4bb8f767-9xcl4_450dc4e8-04b1-40cb-89a7-71d635a80d9d/authorino/0.log"
Oct 31 01:16:28.701143 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:28.701102    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zbvhk_a90600f9-ec16-4ae5-af38-f60501b5ffb6/manager/0.log"
Oct 31 01:16:30.428123 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:30.428089    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r842r_ce1abf08-492e-4e29-a821-6378d706ab91/discovery/0.log"
Oct 31 01:16:30.703500 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:30.703425    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-fslfx_a68e8a72-f823-45d3-ad92-0c597c6433d3/kube-auth-proxy/0.log"
Oct 31 01:16:31.154309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:31.154224    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-7ljch_fa40b2b4-590d-4f48-8363-58137eb2e5bc/istio-proxy/0.log"
Oct 31 01:16:31.707781 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:31.707746    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qn4q8_13732299-1c22-45fb-8d16-b468502d76f3/payload-pre-processing/0.log"
Oct 31 01:16:31.939678 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:31.939649    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 01:16:33.375835 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:33.375804    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-6rd7f_a5baef16-a12f-403b-93f6-32fcb692ed4f/main/0.log"
Oct 31 01:16:33.919867 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:33.919837    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-d2vp6_e2acb604-817d-4688-a5ce-0c8b755ff607/main/0.log"
Oct 31 01:16:34.494032 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:34.494003    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-qf644_c87839e0-6c15-4f16-9683-8242fe26d76b/main/0.log"
Oct 31 01:16:34.780799 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:34.780724    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-7w5wv_88aa52c9-9c13-4b54-b62d-929ff91b26a7/main/0.log"
Oct 31 01:16:44.755556 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:16:44.755522    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:16:56.065825 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:56.065791    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gfzr6_660c1701-2942-48bd-aee9-ff5341e6ffce/global-pull-secret-syncer/0.log"
Oct 31 01:16:56.544355 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:56.544319    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:16:56.737631 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:56.737596    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:16:58.449849 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:58.449764    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:16:58.649974 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:58.649943    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:16:58.959558 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:58.959525    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cprhd_a873a458-b1cc-4b84-9a61-a35b5c783868/konnectivity-agent/0.log"
Oct 31 01:16:59.409582 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:16:59.409488    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-148.us-west-1.compute.internal_3031cca7be5520d5a22eb64a065c3c72/haproxy/0.log"
Oct 31 01:17:03.220881 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:03.220845    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:17:03.456767 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:03.456731    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:17:04.819177 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:04.819135    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4bb8f767-9xcl4_450dc4e8-04b1-40cb-89a7-71d635a80d9d/authorino/0.log"
Oct 31 01:17:05.578929 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.578888    4478 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"]
Oct 31 01:17:05.579254 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579240    4478 cpu_manager.go:414] "containerMap: removing container" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerName="payload-pre-processing"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579256    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerName="payload-pre-processing"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579270    4478 cpu_manager.go:414] "containerMap: removing container" podUID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" containerName="payload-processing"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579276    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" containerName="payload-processing"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579284    4478 cpu_manager.go:414] "containerMap: removing container" podUID="695c4a13-794d-42da-a8ed-a88e0a19610e" containerName="maas-api"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579288    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="695c4a13-794d-42da-a8ed-a88e0a19610e" containerName="maas-api"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579297    4478 cpu_manager.go:414] "containerMap: removing container" podUID="b68e8749-2183-4329-97c7-8f5c9ddb0fbb" containerName="maas-api"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579302    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b68e8749-2183-4329-97c7-8f5c9ddb0fbb" containerName="maas-api"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579307    4478 cpu_manager.go:414] "containerMap: removing container" podUID="1689d832-f216-4ebc-aa4e-650155a07944" containerName="payload-pre-processing"
Oct 31 01:17:05.579309 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579312    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1689d832-f216-4ebc-aa4e-650155a07944" containerName="payload-pre-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579320    4478 cpu_manager.go:414] "containerMap: removing container" podUID="13e1e103-4888-46a9-87c1-b73ae7cad0d1" containerName="payload-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579325    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13e1e103-4888-46a9-87c1-b73ae7cad0d1" containerName="payload-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579349    4478 cpu_manager.go:414] "containerMap: removing container" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerName="payload-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579353    4478 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerName="payload-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579418    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="07ef74ae-1614-4a6c-aa76-a9761c27d98c" containerName="payload-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579425    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdb50a63-262a-4955-9b9b-3a1ea2ad2cc9" containerName="payload-pre-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579432    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="33ea6c3c-609c-4df9-b2ee-c2027372e2c3" containerName="payload-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579438    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="695c4a13-794d-42da-a8ed-a88e0a19610e" containerName="maas-api"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579444    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="b68e8749-2183-4329-97c7-8f5c9ddb0fbb" containerName="maas-api"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579448    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="1689d832-f216-4ebc-aa4e-650155a07944" containerName="payload-pre-processing"
Oct 31 01:17:05.579664 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.579453    4478 memory_manager.go:365] "RemoveStaleState removing state" podUID="13e1e103-4888-46a9-87c1-b73ae7cad0d1" containerName="payload-processing"
Oct 31 01:17:05.582187 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.582172    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.637570 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.637533    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m9n5w\"/\"openshift-service-ca.crt\""
Oct 31 01:17:05.684248 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.684217    4478 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-m9n5w\"/\"kube-root-ca.crt\""
Oct 31 01:17:05.684427 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.684253    4478 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-m9n5w\"/\"default-dockercfg-2hfcm\""
Oct 31 01:17:05.685683 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.685662    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"]
Oct 31 01:17:05.750204 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.750176    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-sys\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.750351 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.750289    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-proc\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.750351 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.750340    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-lib-modules\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.750473 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.750364    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmvzc\" (UniqueName: \"kubernetes.io/projected/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-kube-api-access-hmvzc\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.750473 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.750425    4478 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-podres\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851515 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851441    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-lib-modules\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851515 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851469    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmvzc\" (UniqueName: \"kubernetes.io/projected/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-kube-api-access-hmvzc\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851515 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851502    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-podres\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851524    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-sys\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851583    4478 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-proc\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851585    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-lib-modules\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851596    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-podres\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851657    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-proc\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.851960 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.851660    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-sys\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:05.913105 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:05.913079    4478 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmvzc\" (UniqueName: \"kubernetes.io/projected/4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a-kube-api-access-hmvzc\") pod \"perf-node-gather-daemonset-cdxl2\" (UID: \"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a\") " pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:06.193802 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:06.193778    4478 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:06.282008 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:06.281983    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:17:06.467993 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:06.467925    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:17:06.566361 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:06.566336    4478 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"]
Oct 31 01:17:06.569175 ip-10-0-137-148 kubenswrapper[4478]: W1031 01:17:06.569148    4478 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod4fa7f7c5_f656_4d10_ad89_b90c55eb0a7a.slice/crio-e8606f958d1ca704a6ed67f20599e3d9a349ad1f503fbb50341f4d89ddcad2c3 WatchSource:0}: Error finding container e8606f958d1ca704a6ed67f20599e3d9a349ad1f503fbb50341f4d89ddcad2c3: Status 404 returned error can't find the container with id e8606f958d1ca704a6ed67f20599e3d9a349ad1f503fbb50341f4d89ddcad2c3
Oct 31 01:17:07.077943 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:07.077908    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zbvhk_a90600f9-ec16-4ae5-af38-f60501b5ffb6/manager/0.log"
Oct 31 01:17:07.386530 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:07.386437    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2" event={"ID":"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a","Type":"ContainerStarted","Data":"212e05c93a25e55f3bf943c3a5f16a64e1c05f41c4a01e984356ec87b0dbdcf8"}
Oct 31 01:17:07.386530 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:07.386492    4478 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2" event={"ID":"4fa7f7c5-f656-4d10-ad89-b90c55eb0a7a","Type":"ContainerStarted","Data":"e8606f958d1ca704a6ed67f20599e3d9a349ad1f503fbb50341f4d89ddcad2c3"}
Oct 31 01:17:07.386734 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:07.386540    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:07.504067 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:07.504020    4478 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2" podStartSLOduration=2.504007348 podStartE2EDuration="2.504007348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 01:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 01:17:07.502325234 +0000 UTC m=+4460.518989958" watchObservedRunningTime="2026-10-31 01:17:07.504007348 +0000 UTC m=+4460.520672109"
Oct 31 01:17:09.661593 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:09.661559    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:17:09.857252 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:09.857225    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:17:12.437375 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:12.437344    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654879bb7c-lhq6m_63febb67-eed9-4fda-8104-d832eb9a50a5/manager/0.log"
Oct 31 01:17:12.634083 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:12.634051    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-wtds2_3df50a6f-b1a2-41eb-ae05-a032cca1f2c3/manager/0.log"
Oct 31 01:17:13.401551 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:13.401528    4478 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-m9n5w/perf-node-gather-daemonset-cdxl2"
Oct 31 01:17:14.756375 ip-10-0-137-148 kubenswrapper[4478]: E1031 01:17:14.756341    4478 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-mvljb" containerName="extractor"
Oct 31 01:17:17.239399 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:17.239361    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r842r_ce1abf08-492e-4e29-a821-6378d706ab91/discovery/0.log"
Oct 31 01:17:17.383249 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:17.383212    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-fslfx_a68e8a72-f823-45d3-ad92-0c597c6433d3/kube-auth-proxy/0.log"
Oct 31 01:17:17.442426 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:17.442373    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pp55f_3ff4e5d1-9ced-477b-bd94-b189d25147d7/node-exporter/0.log"
Oct 31 01:17:17.812975 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:17.812943    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-7ljch_fa40b2b4-590d-4f48-8363-58137eb2e5bc/istio-proxy/0.log"
Oct 31 01:17:17.861789 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:17.861768    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pp55f_3ff4e5d1-9ced-477b-bd94-b189d25147d7/kube-rbac-proxy/0.log"
Oct 31 01:17:18.242035 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:18.242006    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qn4q8_13732299-1c22-45fb-8d16-b468502d76f3/payload-pre-processing/0.log"
Oct 31 01:17:18.285560 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:18.285536    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pp55f_3ff4e5d1-9ced-477b-bd94-b189d25147d7/init-textfile/0.log"
Oct 31 01:17:18.451829 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:18.451801    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 01:17:19.879711 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:19.879678    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r842r_ce1abf08-492e-4e29-a821-6378d706ab91/discovery/0.log"
Oct 31 01:17:19.987938 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:19.987906    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-fslfx_a68e8a72-f823-45d3-ad92-0c597c6433d3/kube-auth-proxy/0.log"
Oct 31 01:17:20.422939 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:20.422908    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58c6585447-7ljch_fa40b2b4-590d-4f48-8363-58137eb2e5bc/istio-proxy/0.log"
Oct 31 01:17:20.860256 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:20.860228    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-qn4q8_13732299-1c22-45fb-8d16-b468502d76f3/payload-pre-processing/0.log"
Oct 31 01:17:21.074480 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:21.074444    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-w99g7_c23690f3-509a-470d-8717-5d144235a21a/payload-processing/0.log"
Oct 31 01:17:24.165594 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:24.165567    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4bb8f767-9xcl4_450dc4e8-04b1-40cb-89a7-71d635a80d9d/authorino/0.log"
Oct 31 01:17:24.272131 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:24.272100    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wcc94_9f3c31b2-a6a8-4a46-8f7b-82812f793f38/prometheus-operator/0.log"
Oct 31 01:17:24.683338 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:24.683309    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-wcc94_9f3c31b2-a6a8-4a46-8f7b-82812f793f38/kube-rbac-proxy/0.log"
Oct 31 01:17:25.140507 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:25.140293    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-f292b_202b638d-45b7-4353-ae54-24673443627b/prometheus-operator-admission-webhook/0.log"
Oct 31 01:17:25.498807 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:25.498767    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zbvhk_a90600f9-ec16-4ae5-af38-f60501b5ffb6/manager/0.log"
Oct 31 01:17:27.022370 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:27.022338    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5f4bb8f767-9xcl4_450dc4e8-04b1-40cb-89a7-71d635a80d9d/authorino/0.log"
Oct 31 01:17:28.415750 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:28.415718    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-zbvhk_a90600f9-ec16-4ae5-af38-f60501b5ffb6/manager/0.log"
Oct 31 01:17:30.301870 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:30.301837    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-r842r_ce1abf08-492e-4e29-a821-6378d706ab91/discovery/0.log"
Oct 31 01:17:30.411680 ip-10-0-137-148 kubenswrapper[4478]: I1031 01:17:30.411655    4478 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb564c48d-fslfx_a68e8a72-f823-45d3-ad92-0c597c6433d3/kube-auth-proxy/0.log"